You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/01/04 04:50:34 UTC
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 372 -
Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/372/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 1) Thread[id=29835, name=jetty-launcher-5491-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41) at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244) at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44) at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61) at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
1) Thread[id=29835, name=jetty-launcher-5491-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
at __randomizedtesting.SeedInfo.seed([BF073CDCBB54C00B]:0)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData
Error Message:
Node watch should have fired!
Stack Trace:
java.lang.AssertionError: Node watch should have fired!
at __randomizedtesting.SeedInfo.seed([BF073CDCBB54C00B:9997D38EEFDA0601]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12562 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager
[junit4] 2> 1420100 INFO (SUITE-TestDistribStateManager-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestDistribStateManager_BF073CDCBB54C00B-001/init-core-data-001
[junit4] 2> 1420101 WARN (SUITE-TestDistribStateManager-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 1420101 INFO (SUITE-TestDistribStateManager-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1420102 INFO (SUITE-TestDistribStateManager-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1420104 INFO (TEST-TestDistribStateManager.testListData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testListData
[junit4] 2> 1420104 INFO (TEST-TestDistribStateManager.testListData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
[junit4] 2> 1420104 INFO (TEST-TestDistribStateManager.testListData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testListData
[junit4] 2> 1420106 INFO (TEST-TestDistribStateManager.testCreateMode-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCreateMode
[junit4] 2> 1420106 INFO (TEST-TestDistribStateManager.testCreateMode-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
[junit4] 2> 1420107 INFO (TEST-TestDistribStateManager.testCreateMode-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
[junit4] 2> 1420107 INFO (TEST-TestDistribStateManager.testCreateMode-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCreateMode
[junit4] 2> 1420109 INFO (TEST-TestDistribStateManager.testHasData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testHasData
[junit4] 2> 1420109 INFO (TEST-TestDistribStateManager.testHasData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1420109 INFO (Thread-5015) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1420109 INFO (Thread-5015) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1420111 ERROR (Thread-5015) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1420212 INFO (TEST-TestDistribStateManager.testHasData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.ZkTestServer start zk server on port:56211
[junit4] 2> 1420214 INFO (zkConnectionManagerCallback-3256-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1420214 INFO (TEST-TestDistribStateManager.testHasData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
[junit4] 2> 1420219 ERROR (TEST-TestDistribStateManager.testHasData-seed#[BF073CDCBB54C00B]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1420220 INFO (TEST-TestDistribStateManager.testHasData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:56211 56211
[junit4] 2> 1420785 INFO (Thread-5015) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:56211 56211
[junit4] 2> 1420788 INFO (TEST-TestDistribStateManager.testHasData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testHasData
[junit4] 2> 1420792 INFO (TEST-TestDistribStateManager.testMulti-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testMulti
[junit4] 2> 1420792 INFO (TEST-TestDistribStateManager.testMulti-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.cloud.autoscaling.sim.SimDistribStateManager
[junit4] 2> 1420792 INFO (TEST-TestDistribStateManager.testMulti-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testMulti
[junit4] 2> 1420795 INFO (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testGetSetRemoveData
[junit4] 2> 1420796 INFO (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1420796 INFO (Thread-5016) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1420796 INFO (Thread-5016) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1420799 ERROR (Thread-5016) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1420896 INFO (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.ZkTestServer start zk server on port:53196
[junit4] 2> 1420901 INFO (zkConnectionManagerCallback-3258-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1420901 INFO (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
[junit4] 2> 1420920 INFO (zkConnectionManagerCallback-3260-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1420921 INFO (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.a.s.TestDistribStateManager Using org.apache.solr.client.solrj.impl.ZkDistribStateManager
[junit4] 2> 1425927 ERROR (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1425929 INFO (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53196 53196
[junit4] 2> 1436237 INFO (Thread-5016) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53196 53196
[junit4] 2> 1436239 INFO (TEST-TestDistribStateManager.testGetSetRemoveData-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testGetSetRemoveData
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestDistribStateManager -Dtests.method=testGetSetRemoveData -Dtests.seed=BF073CDCBB54C00B -Dtests.slow=true -Dtests.locale=ru -Dtests.timezone=America/Shiprock -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 15.4s J0 | TestDistribStateManager.testGetSetRemoveData <<<
[junit4] > Throwable #1: java.lang.AssertionError: Node watch should have fired!
[junit4] > at __randomizedtesting.SeedInfo.seed([BF073CDCBB54C00B:9997D38EEFDA0601]:0)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.TestDistribStateManager.testGetSetRemoveData(TestDistribStateManager.java:256)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestDistribStateManager_BF073CDCBB54C00B-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1052, maxMBSortInHeap=7.005888951251702, sim=RandomSimilarity(queryNorm=true): {}, locale=ru, timezone=America/Shiprock
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=159481904,total=387837952
[junit4] 2> NOTE: All tests run in this JVM: [RecoveryZkTest, TestPhraseSuggestions, ImplicitSnitchTest, TestJoin, FieldAnalysisRequestHandlerTest, HdfsDirectoryTest, TestSolrXml, TestSolrCoreSnapshots, TestStressInPlaceUpdates, TestManagedResourceStorage, FacetPivotSmallTest, ScriptEngineTest, TestScoreJoinQPNoScore, TestObjectReleaseTracker, TestJsonRequest, TestUnifiedSolrHighlighter, DateFieldTest, TestInPlaceUpdatesDistrib, TestBackupRepositoryFactory, BigEndianAscendingWordDeserializerTest, DistributedFacetPivotSmallAdvancedTest, TestConfigSets, JavabinLoaderTest, SuggesterWFSTTest, SolrGangliaReporterTest, SubstringBytesRefFilterTest, TestRawTransformer, TestCursorMarkWithoutUniqueKey, TestTolerantSearch, TestCSVResponseWriter, TestPointFields, TestUtils, DeleteNodeTest, TestDeleteCollectionOnDownNodes, HdfsChaosMonkeyNothingIsSafeTest, TestCollectionAPI, SpatialFilterTest, TestSubQueryTransformer, TestNumericRangeQuery32, TestExportWriter, TestReplicaProperties, DataDrivenBlockJoinTest, TestRealTimeGet, TestSmileRequest, RequiredFieldsTest, AutoScalingHandlerTest, TestHalfAndHalfDocValues, TestPolicyCloud, DistributedSpellCheckComponentTest, HdfsDirectoryFactoryTest, DistribJoinFromCollectionTest, TestSolrCloudWithDelegationTokens, SpellCheckCollatorTest, TestPHPSerializedResponseWriter, VMParamsZkACLAndCredentialsProvidersTest, TestConfigReload, ConfigureRecoveryStrategyTest, TestCloudSearcherWarming, HighlighterConfigTest, HdfsCollectionsAPIDistributedZkTest, TestPseudoReturnFields, TestUpdate, ReplicaListTransformerTest, ResourceLoaderTest, SolrInfoBeanTest, SolrCoreMetricManagerTest, TestPolicyCloud, HdfsUnloadDistributedZkTest, BitVectorTest, RankQueryTest, SpellCheckComponentTest, CdcrReplicationDistributedZkTest, CSVRequestHandlerTest, TestDFISimilarityFactory, TestRandomDVFaceting, HdfsBasicDistributedZk2Test, TimeZoneUtilsTest, TestSolrConfigHandlerCloud, TestRecoveryHdfs, TestGenericDistributedQueue, BlockJoinFacetDistribTest, PathHierarchyTokenizerFactoryTest, DistribCursorPagingTest, TestPrepRecovery, AlternateDirectoryTest, StatsReloadRaceTest, PrimUtilsTest, TestSizeLimitedDistributedMap, TestFieldCacheSortRandom, TestReversedWildcardFilterFactory, TimeRoutedAliasUpdateProcessorTest, TestAddFieldRealTimeGet, HighlighterTest, DateMathParserTest, HdfsRestartWhileUpdatingTest, CursorPagingTest, TestSchemaSimilarityResource, HdfsThreadLeakTest, TestQuerySenderListener, TestSolr4Spatial, TestBinaryField, MoveReplicaTest, AnalyticsQueryTest, JvmMetricsTest, DirectSolrConnectionTest, SyncSliceTest, ZkStateReaderTest, JSONWriterTest, ShardRoutingTest, CircularListTest, DocValuesMissingTest, SolrMetricReporterTest, ZkControllerTest, SuggesterTSTTest, TestDownShardTolerantSearch, TestDefaultStatsCache, DistributedExpandComponentTest, TestSolrDeletionPolicy2, TestSchemalessBufferedUpdates, TestGraphMLResponseWriter, UpdateLogTest, UUIDFieldTest, TestQuerySenderNoQuery, ClassificationUpdateProcessorIntegrationTest, XsltUpdateRequestHandlerTest, NumericFieldsTest, DirectoryFactoryTest, SolrTestCaseJ4Test, TestCodecSupport, RemoteQueryErrorTest, SolrJmxReporterCloudTest, OverseerTest, TestExceedMaxTermLength, MultiTermTest, ReturnFieldsTest, TestClusterStateProvider, TestAuthorizationFramework, TestCloudNestedDocsSort, TestReqParamsAPI, ChaosMonkeySafeLeaderWithPullReplicasTest, TestSolrQueryResponse, TestLFUCache, SolrShardReporterTest, ConnectionReuseTest, RegexBoostProcessorTest, TestDistribStateManager]
[junit4] Completed [268/764 (1!)] on J0 in 16.17s, 5 tests, 1 failure <<< FAILURES!
[...truncated 1024 lines...]
[junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
[junit4] 2> 3730283 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_BF073CDCBB54C00B-001/init-core-data-001
[junit4] 2> 3730283 WARN (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
[junit4] 2> 3730284 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 3730286 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 3730322 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_BF073CDCBB54C00B-001/tempDir-001
[junit4] 2> 3730322 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 3730322 INFO (Thread-7561) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 3730323 INFO (Thread-7561) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 3730324 ERROR (Thread-7561) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3730422 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:35273
[junit4] 2> 3730426 INFO (zkConnectionManagerCallback-5494-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730435 INFO (jetty-launcher-5491-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3730436 INFO (jetty-launcher-5491-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3730441 INFO (jetty-launcher-5491-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@47a4c0c4{/solr,null,AVAILABLE}
[junit4] 2> 3730441 INFO (jetty-launcher-5491-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@29c8bae0{/solr,null,AVAILABLE}
[junit4] 2> 3730442 INFO (jetty-launcher-5491-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@188878ae{HTTP/1.1,[http/1.1]}{127.0.0.1:46751}
[junit4] 2> 3730442 INFO (jetty-launcher-5491-thread-2) [ ] o.e.j.s.Server Started @3734811ms
[junit4] 2> 3730443 INFO (jetty-launcher-5491-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46751}
[junit4] 2> 3730443 INFO (jetty-launcher-5491-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3014e0{HTTP/1.1,[http/1.1]}{127.0.0.1:38906}
[junit4] 2> 3730443 INFO (jetty-launcher-5491-thread-1) [ ] o.e.j.s.Server Started @3734812ms
[junit4] 2> 3730443 ERROR (jetty-launcher-5491-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3730443 INFO (jetty-launcher-5491-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 3730443 INFO (jetty-launcher-5491-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3730444 INFO (jetty-launcher-5491-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3730444 INFO (jetty-launcher-5491-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-04T04:11:17.846Z
[junit4] 2> 3730443 INFO (jetty-launcher-5491-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38906}
[junit4] 2> 3730447 ERROR (jetty-launcher-5491-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3730447 INFO (jetty-launcher-5491-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.3.0
[junit4] 2> 3730447 INFO (jetty-launcher-5491-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3730447 INFO (jetty-launcher-5491-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3730447 INFO (jetty-launcher-5491-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-04T04:11:17.849Z
[junit4] 2> 3730451 INFO (zkConnectionManagerCallback-5498-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730452 INFO (jetty-launcher-5491-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3730464 INFO (zkConnectionManagerCallback-5496-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730466 INFO (jetty-launcher-5491-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3730477 INFO (jetty-launcher-5491-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35273/solr
[junit4] 2> 3730481 INFO (jetty-launcher-5491-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35273/solr
[junit4] 2> 3730483 INFO (zkConnectionManagerCallback-5506-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730490 INFO (zkConnectionManagerCallback-5502-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730491 INFO (zkConnectionManagerCallback-5508-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730492 INFO (zkConnectionManagerCallback-5510-thread-1-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730629 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3730629 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38906_solr
[junit4] 2> 3730630 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3730631 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.Overseer Overseer (id=99289600997851142-127.0.0.1:38906_solr-n_0000000000) starting
[junit4] 2> 3730631 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46751_solr
[junit4] 2> 3730633 INFO (zkCallback-5507-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3730633 INFO (zkCallback-5509-thread-1-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3730642 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38906_solr
[junit4] 2> 3730644 INFO (zkCallback-5507-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3730644 INFO (zkCallback-5509-thread-1-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3730669 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 3730670 WARN (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3730670 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 3730672 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 3730677 INFO (jetty-launcher-5491-thread-2-EventThread) [n:127.0.0.1:46751_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 3730690 WARN (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3730711 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 3730711 WARN (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3730712 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 3730713 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 3730715 INFO (jetty-launcher-5491-thread-1-EventThread) [n:127.0.0.1:38906_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 3730738 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46751.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7554e26d
[junit4] 2> 3730742 WARN (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3730744 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46751.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7554e26d
[junit4] 2> 3730744 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46751.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7554e26d
[junit4] 2> 3730745 INFO (jetty-launcher-5491-thread-2) [n:127.0.0.1:46751_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_BF073CDCBB54C00B-001/tempDir-001/node2/.
[junit4] 2> 3730768 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38906.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7554e26d
[junit4] 2> 3730774 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38906.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7554e26d
[junit4] 2> 3730774 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38906.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7554e26d
[junit4] 2> 3730775 INFO (jetty-launcher-5491-thread-1) [n:127.0.0.1:38906_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_BF073CDCBB54C00B-001/tempDir-001/node1/.
[junit4] 2> 3730790 INFO (zkConnectionManagerCallback-5516-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3730808 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
[junit4] 2> 3730813 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
[junit4] 2> 3730833 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
[junit4] 2> 3730834 INFO (qtp1513328013-29797) [ ] o.a.s.s.SolrDispatchFilter Error authenticating
[junit4] 2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
[junit4] 2> at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3730834 WARN (qtp1513328013-29797) [ ] o.e.j.s.ServletHandler /solr/admin/collections
[junit4] 2> org.apache.solr.common.SolrException: Error during request authentication,
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
[junit4] 2> at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> ... 26 more
[junit4] 2> 3730835 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
[junit4] 2> 3730850 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
[junit4] 2> 3730866 INFO (qtp1513328013-29800) [n:127.0.0.1:46751_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3730867 INFO (qtp1513328013-29800) [n:127.0.0.1:46751_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3730867 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
[junit4] 2> 3730885 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testForwarding
[junit4] 2> 3730887 INFO (zkConnectionManagerCallback-5519-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3731370 INFO (zkConnectionManagerCallback-5522-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3731371 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 3731373 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35273/solr ready
[junit4] 2> 3731374 INFO (qtp1165164636-29787) [n:127.0.0.1:38906_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3731376 INFO (OverseerThreadFactory-11758-thread-1-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.CreateCollectionCmd Create collection forwardingCollection
[junit4] 2> 3731489 INFO (OverseerStateUpdate-99289600997851142-127.0.0.1:38906_solr-n_0000000000) [n:127.0.0.1:38906_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"forwardingCollection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"forwardingCollection_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:46751/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3731697 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3731697 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3731806 INFO (zkCallback-5507-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3731806 INFO (zkCallback-5507-thread-2-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3732733 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 3732755 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema [forwardingCollection_shard1_replica_n1] Schema name=test
[junit4] 2> 3732824 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 3732842 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'forwardingCollection_shard1_replica_n1' using configuration from collection forwardingCollection, trusted=true
[junit4] 2> 3732842 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46751.solr.core.forwardingCollection.shard1.replica_n1' (registry 'solr.core.forwardingCollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7554e26d
[junit4] 2> 3732842 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3732842 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [[forwardingCollection_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_BF073CDCBB54C00B-001/tempDir-001/node2/forwardingCollection_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_BF073CDCBB54C00B-001/tempDir-001/node2/./forwardingCollection_shard1_replica_n1/data/]
[junit4] 2> 3732844 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=50, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20503905800116531]
[junit4] 2> 3732906 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3732906 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3732907 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3732907 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3732908 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10635901731026545]
[junit4] 2> 3732908 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@71d2873d[forwardingCollection_shard1_replica_n1] main]
[junit4] 2> 3732910 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
[junit4] 2> 3732913 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 3732929 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 3732929 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
[junit4] 2> 3732929 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588633618697158656
[junit4] 2> 3732931 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
[junit4] 2> 3732931 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
[junit4] 2> 3732931 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
[junit4] 2> 3732931 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
[junit4] 2> 3732932 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
[junit4] 2> 3732932 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
[junit4] 2> 3732932 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
[junit4] 2> 3732933 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
[junit4] 2> 3732934 INFO (searcherExecutor-11763-thread-1-processing-n:127.0.0.1:46751_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] Registered new searcher Searcher@71d2873d[forwardingCollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3732936 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 3732936 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 3732936 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46751/solr/forwardingCollection_shard1_replica_n1/
[junit4] 2> 3732936 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 3732936 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:46751/solr/forwardingCollection_shard1_replica_n1/ has no replicas
[junit4] 2> 3732936 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 3732941 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46751/solr/forwardingCollection_shard1_replica_n1/ shard1
[junit4] 2> 3733044 INFO (zkCallback-5507-thread-2-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3733044 INFO (zkCallback-5507-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3733093 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 3733097 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1400
[junit4] 2> 3733103 INFO (qtp1165164636-29787) [n:127.0.0.1:38906_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 3733199 INFO (zkCallback-5507-thread-2-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3733199 INFO (zkCallback-5507-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3733379 INFO (OverseerCollectionConfigSetProcessor-99289600997851142-127.0.0.1:38906_solr-n_0000000000) [n:127.0.0.1:38906_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 3734106 INFO (qtp1165164636-29787) [n:127.0.0.1:38906_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=2732
[junit4] 2> 3734106 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: forwardingCollection failOnTimeout:true timeout (sec):100
[junit4] 2> 3734107 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[BF073CDCBB54C00B]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: forwardingCollection
[junit4] 2> 3734111 INFO (qtp1513328013-29796) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 3734116 INFO (qtp1165164636-29788) [n:127.0.0.1:38906_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3734121 INFO (qtp1513328013-29797) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 3734122 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testForwarding
[junit4] 2> 3734156 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
[junit4] 2> 3734171 WARN (qtp1513328013-29800) [n:127.0.0.1:46751_solr ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
[junit4] 2> id: invalid user name: "bar"
[junit4] 2>
[junit4] 2> 3734173 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
[junit4] 2> 3734191 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
[junit4] 2> 3734194 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
[junit4] 2> 3734212 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
[junit4] 2> 3734212 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3734212 INFO (qtp1513328013-29793) [n:127.0.0.1:46751_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3734213 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
[junit4] 2> 3734234 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
[junit4] 2> 3748654 WARN (jetty-launcher-5491-thread-2-SendThread(127.0.0.1:35273)) [n:127.0.0.1:46751_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15454ms for sessionid 0x160bf5db55e0005
[junit4] 2> 3748654 WARN (jetty-launcher-5491-thread-2-SendThread(127.0.0.1:35273)) [n:127.0.0.1:46751_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 17906ms for sessionid 0x160bf5db55e0007
[junit4] 2> 3748654 WARN (jetty-launcher-5491-thread-1-SendThread(127.0.0.1:35273)) [n:127.0.0.1:38906_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 17906ms for sessionid 0x160bf5db55e0008
[junit4] 2> 3748655 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160bf5db55e0007, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3748655 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160bf5db55e0005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3748656 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160bf5db55e0008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3748656 WARN (jetty-launcher-5491-thread-1-SendThread(127.0.0.1:35273)) [n:127.0.0.1:38906_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 14537ms for sessionid 0x160bf5db55e0006
[junit4] 2> 3748656 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160bf5db55e0006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3748656 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
[junit4] 2> 3748711 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
[junit4] 2> 3748712 INFO (qtp1513328013-29799) [n:127.0.0.1:46751_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3748712 INFO (qtp1513328013-29799) [n:127.0.0.1:46751_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3748713 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
[junit4] 2> 3748744 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
[junit4] 2> 3748746 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
[junit4] 2> 3748755 INFO (jetty-launcher-5491-thread-2-EventThread) [n:127.0.0.1:46751_solr ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
[junit4] 2> 3748756 WARN (zkConnectionManagerCallback-5522-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d457b91 name: ZooKeeperConnection Watcher:127.0.0.1:35273/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 3748756 WARN (zkConnectionManagerCallback-5522-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 3748760 WARN (zkConnectionManagerCallback-5510-thread-1-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@269dfc66 name: ZooKeeperConnection Watcher:127.0.0.1:35273/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 3748760 WARN (zkConnectionManagerCallback-5510-thread-1-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 3748760 WARN (zkConnectionManagerCallback-5508-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3a98c53f name: ZooKeeperConnection Watcher:127.0.0.1:35273/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 3748761 WARN (zkConnectionManagerCallback-5508-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 3748779 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
[junit4] 2> 3748808 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
[junit4] 2> 3748863 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidRemote
[junit4] 2> 3748869 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[BF073CDCBB54C00B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidRemote
[junit4] 2> 3749004 INFO (jetty-closer-5492-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@188878ae{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 3749005 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1351243310
[junit4] 2> 3749006 INFO (jetty-closer-5492-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@3014e0{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 3749006 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1848684303
[junit4] 2> 3749007 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 3749007 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@c77d8c3: rootName = solr_46751, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1523a657
[junit4] 2> 3749007 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 3749007 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@39ffcb5d: rootName = solr_38906, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6e235901
[junit4] 2> 3749017 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 3749017 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@339d864a: rootName = solr_46751, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@58d8acf0
[junit4] 2> 3749017 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 3749017 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1c13da8d: rootName = solr_38906, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@58d8acf0
[junit4] 2> 3749023 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 3749023 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@63b40c44: rootName = solr_46751, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4061a35b
[junit4] 2> 3749024 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 3749024 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@453764c3: rootName = solr_38906, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4061a35b
[junit4] 2> 3750115 INFO (zkConnectionManagerCallback-5508-thread-1-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3750118 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:46751_solr
[junit4] 2> 3750120 INFO (jetty-closer-5492-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 3750121 INFO (zkCallback-5507-thread-6-processing-n:127.0.0.1:46751_solr) [n:127.0.0.1:46751_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 3750122 INFO (coreCloseExecutor-11768-thread-1) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@48f01630
[junit4] 2> 3750123 INFO (coreCloseExecutor-11768-thread-1) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.forwardingCollection.shard1.replica_n1, tag=1223693872
[junit4] 2> 3750123 INFO (coreCloseExecutor-11768-thread-1) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@40f7bed5: rootName = solr_46751, domain = solr.core.forwardingCollection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.forwardingCollection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@775c2f5c
[junit4] 2> 3750139 INFO (coreCloseExecutor-11768-thread-1) [n:127.0.0.1:46751_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.forwardingCollection.shard1.leader, tag=1223693872
[junit4] 2> 3750144 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160bf5db55e0005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3750146 ERROR (Thread[Thread-7578,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:46751_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 3750154 INFO (jetty-closer-5492-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@47a4c0c4{/solr,null,UNAVAILABLE}
[junit4] 2> 3750305 INFO (zkConnectionManagerCallback-5510-thread-1-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3750307 INFO (zkCallback-5509-thread-5-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 3750309 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:38906_solr
[junit4] 2> 3750311 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 3750312 INFO (jetty-closer-5492-thread-2) [ ] o.a.s.c.Overseer Overseer (id=99289600997851142-127.0.0.1:38906_solr-n_0000000000) closing
[junit4] 2> 3750313 INFO (OverseerStateUpdate-99289600997851142-127.0.0.1:38906_solr-n_0000000000) [n:127.0.0.1:38906_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38906_solr
[junit4] 2> 3750313 INFO (zkCallback-5509-thread-5-processing-n:127.0.0.1:38906_solr) [n:127.0.0.1:38906_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 3750314 WARN (OverseerAutoScalingTriggerThread-99289600997851142-127.0.0.1:38906_solr-n_0000000000) [n:127.0.0.1:38906_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 3750321 ERROR (Thread[Thread-7580,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:38906_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 3750497 INFO (jetty-closer-5492-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@29c8bae0{/solr,null,UNAVAILABLE}
[junit4] 2> 3750500 ERROR (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3750501 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35273 35273
[junit4] 2> 3751911 INFO (Thread-7561) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35273 35273
[junit4] 2> 3751911 WARN (Thread-7561) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2> 2 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/collections/forwardingCollection/state.json
[junit4] 2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
[junit4] 2>
[junit4] 2> 3751937 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[BF073CDCBB54C00B]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Jan 04, 2018 4:11:39 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> Jan 04, 2018 4:11:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
[junit4] 2> 1) Thread[id=29835, name=jetty-launcher-5491-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Jan 04, 2018 4:11:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=29835, name=jetty-launcher-5491-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] 2> 3772190 ERROR (jetty-launcher-5491-thread-2-EventThread) [n:127.0.0.1:46751_solr ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Jan 04, 2018 4:11:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: All leaked threads terminated.
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1985, maxMBSortInHeap=7.513319882251589, sim=RandomSimilarity(queryNorm=true): {}, locale=en-PH, timezone=America/Mendoza
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=252356056,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [RecoveryAfterSoftCommitTest, TestFreeTextSuggestions, BlobRepositoryCloudTest, TestRandomFaceting, TestBlobHandler, BufferStoreTest, DocumentAnalysisRequestHandlerTest, MoreLikeThisHandlerTest, TestLegacyField, TestTolerantUpdateProcessorCloud, TestCopyFieldCollectionResource, DistributedSuggestComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestJsonFacets, TestFastWriter, TestFunctionQuery, TestPostingsSolrHighlighter, DocValuesTest, TestDocBasedVersionConstraints, TestBadConfig, DistributedDebugComponentTest, TestConfigSetProperties, TestDistributedStatsComponentCardinality, SpellPossibilityIteratorTest, SolrJmxReporterTest, TestChildDocTransformer, TestDistributedMissingSort, TestCharFilters, TestCustomDocTransformer, TestSchemaManager, TestRTimerTree, DeleteLastCustomShardedReplicaTest, TestHashPartitioner, HdfsNNFailoverTest, TestCryptoKeys, TestComponentsName, TestManagedResource, TestGeoJSONResponseWriter, TestSSLRandomization, CdcrUpdateLogTest, TestQueryUtils, TestReload, SchemaVersionSpecificBehaviorTest, HdfsAutoAddReplicasIntegrationTest, TestCollationFieldDocValues, TestComputePlanAction, DistributedMLTComponentTest, QueryResultKeyTest, DistribDocExpirationUpdateProcessorTest, TestStressCloudBlindAtomicUpdates, CacheHeaderTest, TestRawResponseWriter, ZkNodePropsTest, TestCoreBackup, OpenCloseCoreStressTest, TestConfigSetsAPIExclusivity, TestTrackingShardHandlerFactory, HdfsRecoveryZkTest, TestMultiWordSynonyms, TestInPlaceUpdatesStandalone, InfixSuggestersTest, SolrCoreTest, TestCrossCoreJoin, SolrMetricsIntegrationTest, BaseCdcrDistributedZkTest, TestClusterStateMutator, BigEndianAscendingWordSerializerTest, TestCollapseQParserPlugin, ShufflingReplicaListTransformerTest, CdcrVersionReplicationTest, RequestLoggingTest, BJQParserTest, TestSimpleTrackingShardHandler, HdfsChaosMonkeySafeLeaderTest, TestSystemIdResolver, V2ApiIntegrationTest, TestRandomCollapseQParserPlugin, TestNodeLostTrigger, TestValueSourceCache, TestLuceneMatchVersion, DeleteStatusTest, TestRandomFlRTGCloud, CoreSorterTest, SystemInfoHandlerTest, TestFastOutputStream, TestSolrCloudWithKerberosAlt, TestFieldCacheSort, TestWordDelimiterFilterFactory, SpatialRPTFieldTypeTest, TolerantUpdateProcessorTest, TestDocSet, HighlighterMaxOffsetTest, SimplePostToolTest, HdfsTlogReplayBufferedWhileIndexingTest, BasicFunctionalityTest, TestUniqueKeyFieldResource, HdfsWriteToMultipleCollectionsTest, TestShardHandlerFactory, TestReloadDeadlock, SynonymTokenizerTest, MissingSegmentRecoveryTest, MergeStrategyTest, SolrMetricManagerTest, BasicAuthIntegrationTest, TestCloudPivotFacet, RuleEngineTest, TestMacros, ShardSplitTest, FileUtilsTest, EnumFieldTest, SolrGraphiteReporterTest, AutoAddReplicasPlanActionTest, FileBasedSpellCheckerTest, TestLeaderElectionWithEmptyReplica, TestBM25SimilarityFactory, BadComponentTest, TestSolrCloudSnapshots, TestUseDocValuesAsStored, TestJavabinTupleStreamParser, UpdateParamsTest, CursorMarkTest, TestSolrConfigHandler, ClassificationUpdateProcessorFactoryTest, InfoHandlerTest, PreAnalyzedFieldManagedSchemaCloudTest, PluginInfoTest, TestDistributedGrouping, TestConfig, ReplaceNodeTest, SolrCloudReportersTest, OverseerRolesTest, SolrIndexSplitterTest, OpenExchangeRatesOrgProviderTest, TestComplexPhraseQParserPlugin, TestExecutePlanAction, TestPerFieldSimilarity, DistributedFacetSimpleRefinementLongTailTest, TestSQLHandlerNonCloud, ChaosMonkeyNothingIsSafeWithPullReplicasTest, TestSubQueryTransformerCrossCore, TestGraphTermsQParserPlugin, SimpleFacetsTest, TestJettySolrRunner, TestLargeCluster, OutOfBoxZkACLAndCredentialsProvidersTest, TermVectorComponentTest, TestDocumentBuilder, MoveReplicaHDFSTest, TestNonDefinedSimilarityFactory, TestManagedSchemaThreadSafety, BasicDistributedZk2Test, DateRangeFieldTest, PingRequestHandlerTest, ShardRoutingCustomTest, TestConfigOverlay, TestImpersonationWithHadoopAuth, NodeLostTriggerTest, EchoParamsTest, TestAuthenticationFramework, ExternalFileFieldSortTest, SparseHLLTest, TestFieldCacheVsDocValues, TestSimpleQParserPlugin, DocValuesMultiTest, TestSolrQueryParser, HdfsSyncSliceTest, DirectSolrSpellCheckerTest, LegacyCloudClusterPropTest, DistributedFacetExistsSmallTest, BlockJoinFacetRandomTest, CachingDirectoryFactoryTest, ClassificationUpdateProcessorTest, TestSha256AuthenticationProvider, TestExactStatsCache, BlockCacheTest, TestMaxScoreQueryParser, TestRemoteStreaming, PolyFieldTest, ExplicitHLLTest, TestElisionMultitermQuery, SampleTest, SpellingQueryConverterTest, BadCopyFieldTest, TestRestoreCore, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, ClusterStateUpdateTest, ZkSolrClientTest, ZkCLITest, TestGroupingSearch, QueryElevationComponentTest, BadIndexSchemaTest, ConvertedLegacyTest, TestSort, TestFiltering, TestLazyCores, DirectUpdateHandlerTest, TestIndexSearcher, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, CoreAdminHandlerTest, TestFoldingMultitermQuery, SuggesterTest, TestStressLucene, TestTrie, TestCSVLoader, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, DistanceFunctionTest, TestSolrDeletionPolicy1, DebugComponentTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryTypes, PrimitiveFieldTypeTest, TermsComponentTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, DocumentBuilderTest, TestIndexingPerformance, FastVectorHighlighterTest, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, SolrPluginUtilsTest, TestJmxIntegration, TestCollationField, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, QueryParsingTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, TestPartialUpdateDeduplication, SearchHandlerTest, SOLR749Test, TestSolrIndexConfig, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, TestStressRecovery, TestMergePolicyConfig, TestSolrCoreProperties, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, AssignBackwardCompatibilityTest, AsyncCallRequestStatusResponseTest, CleanupOldIndexTest, CollectionTooManyReplicasTest, CollectionsAPISolrJTest, CustomCollectionTest, DeleteReplicaTest, DistributedVersionInfoTest, HttpPartitionTest, LeaderFailureAfterFreshStartTest, LeaderInitiatedRecoveryOnCommitTest, MultiThreadedOCPTest, OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, OverseerTaskQueueTest, SSLMigrationTest, SharedFSAutoReplicaFailoverTest, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestClusterProperties, TestCollectionsAPIViaSolrCloudCluster, TestConfigSetsAPI, TestLeaderElectionZkExpiry, TestLocalFSCloudBackupRestore, TestOnReconnectListenerSupport, TestPullReplica, TestRebalanceLeaders, TestSegmentSorting, TestShortCircuitedRequests, TestSolrCloudWithSecureImpersonation]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=BF073CDCBB54C00B -Dtests.slow=true -Dtests.locale=en-PH -Dtests.timezone=America/Mendoza -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | TestSolrCloudWithSecureImpersonation (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
[junit4] > 1) Thread[id=29835, name=jetty-launcher-5491-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] > at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] > at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] > at __randomizedtesting.SeedInfo.seed([BF073CDCBB54C00B]:0)
[junit4] Completed [575/764 (2!)] on J1 in 41.93s, 12 tests, 1 error <<< FAILURES!
[...truncated 48722 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 377 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/377/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testCooldown
Error Message:
Stack Trace:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([A494A0B444D55030:952ACD503A7F25C2]:0)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testCooldown(TestTriggerIntegration.java:1084)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12207 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_A494A0B444D55030-001/init-core-data-001
[junit4] 2> 1296227 WARN (SUITE-TestTriggerIntegration-seed#[A494A0B444D55030]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=32 numCloses=32
[junit4] 2> 1296227 INFO (SUITE-TestTriggerIntegration-seed#[A494A0B444D55030]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1296229 INFO (SUITE-TestTriggerIntegration-seed#[A494A0B444D55030]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1296229 INFO (SUITE-TestTriggerIntegration-seed#[A494A0B444D55030]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1296230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1296231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 1296231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1296231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 1296231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 1296231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296231 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
[junit4] 2> 1296231 DEBUG (ScheduledTrigger-2144-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296234 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
[junit4] 2> 1296234 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 1296234 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1296234 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1296235 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1296236 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1296236 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 1296236 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1296236 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 1296236 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1296236 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296236 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1296236 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296265 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296302 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296325 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296339 DEBUG (simCloudManagerPool-2146-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1296339 DEBUG (simCloudManagerPool-2146-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 1296340 DEBUG (simCloudManagerPool-2146-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1296340 DEBUG (simCloudManagerPool-2146-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1296340 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1296340 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1296340 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296341 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1296341 DEBUG (ScheduledTrigger-2147-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 1296350 DEBUG (ScheduledTrigger-2147-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 1296350 DEBUG (ScheduledTrigger-2147-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10006_solr
[junit4] 2> 1296361 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 1296361 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10006_solr
[junit4] 2> 1296372 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 1296383 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 1296383 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10006_solr]
[junit4] 2> 1296383 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaae4689ea4bdT9cexyk69jg0ohfkc8shjh1tvq",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":3002647751599293,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002647751599293],
[junit4] 2> "nodeNames":["127.0.0.1:10006_solr"]}}
[junit4] 2> 1296394 DEBUG (AutoscalingActionExecutor-2148-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaae4689ea4bdT9cexyk69jg0ohfkc8shjh1tvq",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":3002647751599293,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002647751599293],
[junit4] 2> "_enqueue_time_":3002648827921343,
[junit4] 2> "nodeNames":["127.0.0.1:10006_solr"]}}
[junit4] 2> 1296403 DEBUG (simCloudManagerPool-2146-thread-3) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1296403 DEBUG (simCloudManagerPool-2146-thread-3) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 1296403 DEBUG (simCloudManagerPool-2146-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr]
[junit4] 2> 1296404 DEBUG (simCloudManagerPool-2146-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr]
[junit4] 2> 1296404 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1296404 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296404 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1296414 DEBUG (AutoscalingActionExecutor-2148-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 991 ms for event id=aaae4689ea4bdT9cexyk69jg0ohfkc8shjh1tvq
[junit4] 2> 1296414 DEBUG (ScheduledTrigger-2147-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 1296414 DEBUG (ScheduledTrigger-2147-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 1296440 DEBUG (ScheduledTrigger-2147-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 1
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 1
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10006_solr
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 2
[junit4] 2> 1296447 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1296448 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger
[junit4] 2> 1296448 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1296448 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1296452 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testEventQueue
[junit4] 2> 1296452 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 1296452 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 1296453 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1296453 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296453 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 1296453 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10006_solr
[junit4] 2> 1296453 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1296440 DEBUG (ScheduledTrigger-2147-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 1296453 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1296453 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1296454 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1296454 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 1296454 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1296454 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
[junit4] 2> 1296454 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 1296455 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296455 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 1296456 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296477 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296501 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296521 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296542 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296559 DEBUG (simCloudManagerPool-2149-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1296559 DEBUG (simCloudManagerPool-2149-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 1296559 DEBUG (simCloudManagerPool-2149-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1296560 INFO (simCloudManagerPool-2149-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventQueueAction instantiated
[junit4] 2> 1296560 DEBUG (simCloudManagerPool-2149-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1296560 DEBUG (simCloudManagerPool-2149-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventQueueAction}], enabled=true}
[junit4] 2> 1296560 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 1296560 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1296560 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 1296560 DEBUG (ScheduledTrigger-2150-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296560 DEBUG (ScheduledTrigger-2150-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1296560 DEBUG (ScheduledTrigger-2150-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10008_solr at time 3002657704495093
[junit4] 2> 1296566 DEBUG (ScheduledTrigger-2150-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1296581 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296581 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1296581 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10008_solr] added at times [3002657704495093], now=3002658749355443
[junit4] 2> 1296581 DEBUG (ScheduledTrigger-2150-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002657704495093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002657704495093],
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"]}}
[junit4] 2> 1296582 DEBUG (AutoscalingActionExecutor-2151-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002657704495093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002657704495093],
[junit4] 2> "_enqueue_time_":3002658756249143,
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"]}}
[junit4] 2> 1296582 INFO (AutoscalingActionExecutor-2151-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration -- event: {
[junit4] 2> "id":"aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002657704495093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002657704495093],
[junit4] 2> "_enqueue_time_":3002658756249143,
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"]}}
[junit4] 2> 1296583 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1296583 DEBUG (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1296583 WARN (AutoscalingActionExecutor-2151-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Exception executing actions
[junit4] 2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger node_added_trigger1 has been closed.
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.dequeue(ScheduledTriggers.java:494)
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:306)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1296583 DEBUG (AutoscalingActionExecutor-2151-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 55 ms for event id=aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt
[junit4] 2> 1296583 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1296584 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 1296584 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1296584 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventQueueAction instantiated
[junit4] 2> 1296584 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1296584 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventQueueAction}], enabled=true}
[junit4] 2> 1296584 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
[junit4] 2> 1296584 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 1296585 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10007_solr
[junit4] 2> 1296585 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1296585 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 1296585 DEBUG (ScheduledTrigger-2153-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296585 DEBUG (ScheduledTrigger-2153-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10007_solr
[junit4] 2> 1296585 DEBUG (ScheduledTrigger-2153-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002657704495093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002657704495093],
[junit4] 2> "_dequeue_time_":3002658935684643,
[junit4] 2> "_enqueue_time_":3002658756249143,
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"],
[junit4] 2> "replaying":true}}
[junit4] 2> 1296586 DEBUG (ScheduledTrigger-2153-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296586 DEBUG (ScheduledTrigger-2153-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1296587 DEBUG (AutoscalingActionExecutor-2154-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002657704495093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002657704495093],
[junit4] 2> "_dequeue_time_":3002658935684643,
[junit4] 2> "_enqueue_time_":3002658756249143,
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"],
[junit4] 2> "replaying":true}}
[junit4] 2> 1296587 INFO (AutoscalingActionExecutor-2154-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration -- event: {
[junit4] 2> "id":"aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002657704495093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002657704495093],
[junit4] 2> "_dequeue_time_":3002658935684643,
[junit4] 2> "_enqueue_time_":3002658756249143,
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"],
[junit4] 2> "replaying":true}}
[junit4] 2> 1296594 DEBUG (AutoscalingActionExecutor-2154-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 346 ms for event id=aaae6b9dbc7f5T9cexyk69jg0ohfkc8shjh1tvt
[junit4] 2> 1296607 DEBUG (ScheduledTrigger-2153-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296608 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296608 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1296637 DEBUG (ScheduledTrigger-2153-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296642 DEBUG (ScheduledTrigger-2153-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296642 DEBUG (ScheduledTrigger-2153-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1296663 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296664 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1296664 DEBUG (ScheduledTrigger-2153-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testEventQueue
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 2
[junit4] 2> 1296684 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296684 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10007_solr
[junit4] 2> 1296684 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 1
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 3
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger1
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 2
[junit4] 2> 1296684 INFO (TEST-TestTriggerIntegration.testEventQueue-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1296705 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1296705 DEBUG (ScheduledTrigger-2153-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296705 DEBUG (ScheduledTrigger-2153-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1296708 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCooldown
[junit4] 2> 1296708 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 1296708 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10007_solr
[junit4] 2> 1296708 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1296709 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1296710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1296710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 1296710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1296710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion -1
[junit4] 2> 1296710 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 1296711 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296711 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 1296711 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296717 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1296731 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296751 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296773 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296825 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296826 DEBUG (simCloudManagerPool-2155-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1296826 DEBUG (simCloudManagerPool-2155-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 1296827 DEBUG (simCloudManagerPool-2155-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1296827 DEBUG (simCloudManagerPool-2155-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1296827 DEBUG (simCloudManagerPool-2155-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1296827 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 1296827 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1296827 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
[junit4] 2> 1296827 DEBUG (ScheduledTrigger-2156-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1296827 DEBUG (ScheduledTrigger-2156-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1296833 DEBUG (simCloudManagerPool-2155-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1296841 DEBUG (simCloudManagerPool-2155-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 1296841 DEBUG (simCloudManagerPool-2155-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1296841 DEBUG (simCloudManagerPool-2155-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1296841 DEBUG (simCloudManagerPool-2155-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1296841 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 1296841 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
[junit4] 2> 1296846 DEBUG (ScheduledTrigger-2156-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1296848 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1296848 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1296848 DEBUG (ScheduledTrigger-2156-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10009_solr at time 3002672093959943
[junit4] 2> 1296893 DEBUG (ScheduledTrigger-2156-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1296893 DEBUG (ScheduledTrigger-2156-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 1296893 DEBUG (ScheduledTrigger-2156-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1296893 DEBUG (ScheduledTrigger-2156-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10009_solr] added at times [3002672093959943], now=3002674356077593
[junit4] 2> 1296894 DEBUG (ScheduledTrigger-2156-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaaea13899707T9cexyk69jg0ohfkc8shjh1tvx",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":3002672093959943,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002672093959943],
[junit4] 2> "nodeNames":["127.0.0.1:10009_solr"]}}
[junit4] 2> 1296894 DEBUG (AutoscalingActionExecutor-2157-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaaea13899707T9cexyk69jg0ohfkc8shjh1tvx",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":3002672093959943,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002672093959943],
[junit4] 2> "_enqueue_time_":3002674367186143,
[junit4] 2> "nodeNames":["127.0.0.1:10009_solr"]}}
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCooldown
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 3
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 3
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 1
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_cooldown_trigger
[junit4] 2> 1296930 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1296932 DEBUG (AutoscalingActionExecutor-2157-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 1874 ms for event id=aaaea13899707T9cexyk69jg0ohfkc8shjh1tvx
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestTriggerIntegration -Dtests.method=testCooldown -Dtests.seed=A494A0B444D55030 -Dtests.slow=true -Dtests.locale=ar-YE -Dtests.timezone=America/Bahia -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.25s J0 | TestTriggerIntegration.testCooldown <<<
[junit4] > Throwable #1: java.lang.NullPointerException
[junit4] > at __randomizedtesting.SeedInfo.seed([A494A0B444D55030:952ACD503A7F25C2]:0)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testCooldown(TestTriggerIntegration.java:1084)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1296947 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
[junit4] 2> 1296947 DEBUG (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 1296948 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1296948 DEBUG (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1296949 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1296949 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1296949 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
[junit4] 2> 1296949 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1296949 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion -1
[junit4] 2> 1296949 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
[junit4] 2> 1296949 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1296949 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
[junit4] 2> 1296949 DEBUG (ScheduledTrigger-2159-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296970 DEBUG (ScheduledTrigger-2159-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1296991 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297011 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297034 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297054 DEBUG (simCloudManagerPool-2158-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1297054 DEBUG (simCloudManagerPool-2158-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
[junit4] 2> 1297054 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297055 DEBUG (simCloudManagerPool-2158-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297055 DEBUG (simCloudManagerPool-2158-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297055 DEBUG (simCloudManagerPool-2158-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1297055 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
[junit4] 2> 1297055 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1297055 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
[junit4] 2> 1297055 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297055 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1297056 DEBUG (simCloudManagerPool-2158-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1297056 DEBUG (simCloudManagerPool-2158-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
[junit4] 2> 1297056 DEBUG (simCloudManagerPool-2158-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297056 DEBUG (simCloudManagerPool-2158-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1297056 DEBUG (simCloudManagerPool-2158-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297056 DEBUG (simCloudManagerPool-2158-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297056 DEBUG (simCloudManagerPool-2158-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1297061 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
[junit4] 2> 1297061 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1297062 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
[junit4] 2> 1297062 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297062 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1297075 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1297075 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297075 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297075 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10010_solr at time 3002683462120193
[junit4] 2> 1297075 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:10010_solr] added at times [3002683462120193], now=3002683463929043
[junit4] 2> 1297076 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaaecb921f301T9cexyk69jg0ohfkc8shjh1tw0",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002683462120193,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683462120193],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297078 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaaecb921f301T9cexyk69jg0ohfkc8shjh1tw0",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":3002683462120193,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683462120193],
[junit4] 2> "_enqueue_time_":3002683469805243,
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297079 INFO (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger1
[junit4] 2> 1297080 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 105 ms for event id=aaaecb921f301T9cexyk69jg0ohfkc8shjh1tw0
[junit4] 2> 1297083 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297083 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297083 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10010_solr at time 3002683859922093
[junit4] 2> 1297083 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10010_solr] added at times [3002683859922093], now=3002683863215243
[junit4] 2> 1297084 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
[junit4] 2> "id":"aaaecd0d7ecadT9cexyk69jg0ohfkc8shjh1tw3",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":3002683859922093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683859922093],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297096 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1297099 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297099 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297110 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297110 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297110 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10010_solr] added at times [3002683859922093], now=3002685195938593
[junit4] 2> 1297110 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
[junit4] 2> "id":"aaaecd0d7ecadT9cexyk69jg0ohfkc8shjh1tw5",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":3002683859922093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683859922093],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297123 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1297123 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297123 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297133 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297133 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297133 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10010_solr] added at times [3002683859922093], now=3002686358596293
[junit4] 2> 1297133 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
[junit4] 2> "id":"aaaecd0d7ecadT9cexyk69jg0ohfkc8shjh1tw7",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":3002683859922093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683859922093],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297143 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1297143 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297144 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297159 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297159 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297159 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10010_solr] added at times [3002683859922093], now=3002687626107643
[junit4] 2> 1297159 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
[junit4] 2> "id":"aaaecd0d7ecadT9cexyk69jg0ohfkc8shjh1tw9",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":3002683859922093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683859922093],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297170 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1297170 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297170 DEBUG (ScheduledTrigger-2159-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297183 DEBUG (ScheduledTrigger-2159-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297183 DEBUG (ScheduledTrigger-2159-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297183 DEBUG (ScheduledTrigger-2159-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:10010_solr] added at times [3002683859922093], now=3002688859453943
[junit4] 2> 1297183 DEBUG (ScheduledTrigger-2159-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaaecd0d7ecadT9cexyk69jg0ohfkc8shjh1twb",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":3002683859922093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683859922093],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297185 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaaecd0d7ecadT9cexyk69jg0ohfkc8shjh1twb",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":3002683859922093,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002683859922093],
[junit4] 2> "_enqueue_time_":3002688866692293,
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297185 INFO (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration last action at 3002683619055093 time = 3002688942761043
[junit4] 2> 1297185 INFO (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_added_trigger2
[junit4] 2> 1297191 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 282 ms for event id=aaaecd0d7ecadT9cexyk69jg0ohfkc8shjh1twb
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1297191 DEBUG (simCloudManagerPool-2158-thread-11) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297192 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
[junit4] 2> 1297192 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1297192 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
[junit4] 2> 1297193 DEBUG (ScheduledTrigger-2159-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 3
[junit4] 2> 1297194 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1297194 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 17
[junit4] 2> 1297194 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297194 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1297194 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297194 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297199 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 1297199 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297199 DEBUG (simCloudManagerPool-2158-thread-12) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297200 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
[junit4] 2> 1297200 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1297200 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
[junit4] 2> 1297201 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with currently live nodes: 2
[junit4] 2> 1297201 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10010_solr
[junit4] 2> 1297201 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10010_solr]
[junit4] 2> 1297202 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaaee30b679dbT9cexyk69jg0ohfkc8shjh1twe",
[junit4] 2> "source":"node_lost_trigger2",
[junit4] 2> "eventTime":3002689763310043,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002689763310043],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297203 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaaee30b679dbT9cexyk69jg0ohfkc8shjh1twe",
[junit4] 2> "source":"node_lost_trigger2",
[junit4] 2> "eventTime":3002689763310043,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002689763310043],
[junit4] 2> "_enqueue_time_":3002689770006293,
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297203 INFO (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_lost_trigger2
[junit4] 2> 1297204 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 61 ms for event id=aaaee30b679dbT9cexyk69jg0ohfkc8shjh1twe
[junit4] 2> 1297210 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297210 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1297210 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297211 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10010_solr
[junit4] 2> 1297211 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297211 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1297284 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 2
[junit4] 2> 1297284 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10010_solr
[junit4] 2> 1297284 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10010_solr]
[junit4] 2> 1297285 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
[junit4] 2> "id":"aaaef2812e183T9cexyk69jg0ohfkc8shjh1twh",
[junit4] 2> "source":"node_lost_trigger1",
[junit4] 2> "eventTime":3002693913338243,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002693913338243],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297285 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger processor for lost nodes: [127.0.0.1:10010_solr] is not ready, will try later
[junit4] 2> 1297285 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger2 with currently live nodes: 2
[junit4] 2> 1297286 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 1297286 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1297286 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297286 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 1297286 DEBUG (ScheduledTrigger-2159-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1297309 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger1 with currently live nodes: 2
[junit4] 2> 1297309 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10010_solr]
[junit4] 2> 1297309 DEBUG (ScheduledTrigger-2159-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"aaaef2812e183T9cexyk69jg0ohfkc8shjh1twj",
[junit4] 2> "source":"node_lost_trigger1",
[junit4] 2> "eventTime":3002693913338243,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002693913338243],
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297310 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"aaaef2812e183T9cexyk69jg0ohfkc8shjh1twj",
[junit4] 2> "source":"node_lost_trigger1",
[junit4] 2> "eventTime":3002693913338243,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[3002693913338243],
[junit4] 2> "_enqueue_time_":3002695143747443,
[junit4] 2> "nodeNames":["127.0.0.1:10010_solr"]}}
[junit4] 2> 1297310 INFO (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration last action at 3002689832865643 time = 3002695213637943
[junit4] 2> 1297310 INFO (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration action executed from node_lost_trigger1
[junit4] 2> 1297311 DEBUG (AutoscalingActionExecutor-2160-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 51 ms for event id=aaaef2812e183T9cexyk69jg0ohfkc8shjh1twj
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testTriggerThrottling
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 2
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10010_solr
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 4
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 13
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger2
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - IGNORED 4
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - IGNORED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger2
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1297312 INFO (TEST-TestTriggerIntegration.testTriggerThrottling-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1297330 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 1297330 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 18
[junit4] 2> 1297330 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10010_solr
[junit4] 2> 1297330 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1297330 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1297335 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1297337 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1297337 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 19
[junit4] 2> 1297337 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297337 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion -1
[junit4] 2> 1297337 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 19
[junit4] 2> 1297338 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1297338 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion 19
[junit4] 2> 1297338 DEBUG (ScheduledTrigger-2162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297361 DEBUG (ScheduledTrigger-2162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297381 DEBUG (ScheduledTrigger-2162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297401 DEBUG (ScheduledTrigger-2162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297422 DEBUG (ScheduledTrigger-2162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297442 DEBUG (simCloudManagerPool-2161-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1297442 DEBUG (simCloudManagerPool-2161-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 20
[junit4] 2> 1297443 DEBUG (ScheduledTrigger-2162-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1297443 DEBUG (simCloudManagerPool-2161-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297443 DEBUG (simCloudManagerPool-2161-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10005_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1297443 DEBUG (simCloudManagerPool-2161-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1297445 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 20
[junit4] 2> 1297445 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1297445 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 20, lastZnodeVersion 20
[junit4] 2> 1297445 DEBUG (ScheduledTrigger-2162-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1297445 DEBUG (ScheduledTrigger-2162-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1297464 DEBUG (ScheduledTrigger-2162-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1297466 DEBUG (ScheduledTrigger-2162-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1297466 DEBUG (ScheduledTrigger-2162-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1297466 DEBUG (ScheduledTrigger-2162-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10011_solr at time 3002702982980193
[junit4] 2> 1297484 DEBUG (ScheduledTrigg
[...truncated too long message...]
[junit4] 2> "properties":{
[junit4] 2> "node":{
[junit4] 2> "127.0.0.1:10018_solr":250.0,
[junit4] 2> "127.0.0.1:10019_solr":250.0},
[junit4] 2> "replica":[
[junit4] 2> "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":123450000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10018_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
[junit4] 2> "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"INDEX.sizeInBytes\":123450000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10019_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
[junit4] 2> "collection":{"collection1":500.0},
[junit4] 2> "shard":{"collection1":{"shard1":500.0}}}}
[junit4] 2> 1299533 DEBUG (ScheduledTrigger-2189-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1299534 DEBUG (ScheduledTrigger-2189-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Cooldown period - rejecting event: {
[junit4] 2> "id":"aab088034e41fT9cexyk69jg0ohfkc8shjh1txi",
[junit4] 2> "source":"search_rate_trigger",
[junit4] 2> "eventTime":3002802766144543,
[junit4] 2> "eventType":"SEARCHRATE",
[junit4] 2> "properties":{
[junit4] 2> "node":{
[junit4] 2> "127.0.0.1:10018_solr":250.0,
[junit4] 2> "127.0.0.1:10019_solr":250.0},
[junit4] 2> "replica":[
[junit4] 2> "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":123450000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10018_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
[junit4] 2> "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"INDEX.sizeInBytes\":123450000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10019_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
[junit4] 2> "collection":{"collection1":500.0},
[junit4] 2> "shard":{"collection1":{"shard1":500.0}}}}
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSearchRate
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 0
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: [collection1]
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 1
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 1
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 2
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * collection1 2
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - active 2
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - CREATE 1
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 4
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: search_rate_trigger
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - IGNORED 2
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1299537 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[A494A0B444D55030]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1299538 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1299539 DEBUG (SUITE-TestTriggerIntegration-seed#[A494A0B444D55030]-worker) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_A494A0B444D55030-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1165, maxMBSortInHeap=6.558980622339023, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-YE, timezone=America/Bahia
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=105337192,total=386097152
[junit4] 2> NOTE: All tests run in this JVM: [FieldAnalysisRequestHandlerTest, TestDownShardTolerantSearch, TestMaxTokenLenTokenizer, SuggesterWFSTTest, SolrXmlInZkTest, TestOmitPositions, BasicAuthIntegrationTest, DeleteShardTest, SolrMetricsIntegrationTest, TestGraphTermsQParserPlugin, CursorMarkTest, TestFiltering, SolrTestCaseJ4Test, HighlighterConfigTest, XsltUpdateRequestHandlerTest, TestCloudJSONFacetJoinDomain, TestFuzzyAnalyzedSuggestions, TestDistributedStatsComponentCardinality, TestReRankQParserPlugin, VMParamsZkACLAndCredentialsProvidersTest, DistanceUnitsTest, TestCustomDocTransformer, MinimalSchemaTest, PeerSyncReplicationTest, TestSolrCloudWithDelegationTokens, BJQParserTest, TestCloudSearcherWarming, TestSubQueryTransformerCrossCore, TestReloadDeadlock, UUIDFieldTest, LeaderElectionTest, TestSimpleQParserPlugin, CdcrBidirectionalTest, SmileWriterTest, TestComplexPhraseQParserPlugin, UnloadDistributedZkTest, DirectSolrConnectionTest, BadCopyFieldTest, TestJoin, ShowFileRequestHandlerTest, DisMaxRequestHandlerTest, DistributedQueryElevationComponentTest, TestInitParams, TestReplicaProperties, TestRawResponseWriter, MetricsHandlerTest, TimeZoneUtilsTest, SolrCLIZkUtilsTest, TestEmbeddedSolrServerSchemaAPI, TestIBSimilarityFactory, StatsComponentTest, ConvertedLegacyTest, CSVRequestHandlerTest, TestFieldCache, SpellCheckComponentTest, DistributedFacetPivotSmallTest, OverseerTest, ScriptEngineTest, SpellCheckCollatorWithCollapseTest, ConfigSetsAPITest, URLClassifyProcessorTest, TestXIncludeConfig, TestSizeLimitedDistributedMap, HttpTriggerListenerTest, TestJsonFacets, BinaryUpdateRequestHandlerTest, TestJmxIntegration, SliceStateTest, TestLegacyFieldCache, CoreMergeIndexesAdminHandlerTest, ZkControllerTest, ClusterStateTest, TestTrackingShardHandlerFactory, DistributedDebugComponentTest, RegexBytesRefFilterTest, ExecutePlanActionTest, CreateCollectionCleanupTest, TestPolicyCloud, PreAnalyzedFieldManagedSchemaCloudTest, TestReqParamsAPI, TestSchemaVersionResource, RollingRestartTest, OverriddenZkACLAndCredentialsProvidersTest, TestQuerySenderNoQuery, DateMathParserTest, TestFieldCacheVsDocValues, TestWriterPerf, DefaultValueUpdateProcessorTest, SparseHLLTest, IndexBasedSpellCheckerTest, TestTolerantSearch, HdfsCollectionsAPIDistributedZkTest, StressHdfsTest, EnumFieldTest, TestStreamBody, ChaosMonkeyNothingIsSafeTest, TestRangeQuery, ExplicitHLLTest, TestTriggerIntegration]
[junit4] Completed [174/765 (1!)] on J0 in 3.34s, 11 tests, 1 error <<< FAILURES!
[...truncated 50127 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 376 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/376/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
Error Message:
3 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 1) Thread[id=11305, name=jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 2) Thread[id=11306, name=jetty-launcher-4197-thread-2-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 3) Thread[id=11235, name=jetty-launcher-4197-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41) at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244) at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44) at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61) at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
1) Thread[id=11305, name=jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
2) Thread[id=11306, name=jetty-launcher-4197-thread-2-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
3) Thread[id=11235, name=jetty-launcher-4197-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
at __randomizedtesting.SeedInfo.seed([A5CD4223B6915CDB]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=11305, name=jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=11305, name=jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
at __randomizedtesting.SeedInfo.seed([A5CD4223B6915CDB]:0)
Build Log:
[...truncated 12409 lines...]
[junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_A5CD4223B6915CDB-001/init-core-data-001
[junit4] 2> 1055859 WARN (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
[junit4] 2> 1055860 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1055861 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1055861 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1055887 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_A5CD4223B6915CDB-001/tempDir-001
[junit4] 2> 1055887 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1055887 INFO (Thread-2057) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1055887 INFO (Thread-2057) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1055889 ERROR (Thread-2057) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1055990 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:41494
[junit4] 2> 1055993 INFO (zkConnectionManagerCallback-4200-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1055996 INFO (jetty-launcher-4197-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1055997 INFO (jetty-launcher-4197-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1055998 INFO (jetty-launcher-4197-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b435cd4{/solr,null,AVAILABLE}
[junit4] 2> 1055998 INFO (jetty-launcher-4197-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@604ca546{/solr,null,AVAILABLE}
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1c2c5c1{HTTP/1.1,[http/1.1]}{127.0.0.1:63867}
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@34617391{HTTP/1.1,[http/1.1]}{127.0.0.1:45454}
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-1) [ ] o.e.j.s.Server Started @1059886ms
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-2) [ ] o.e.j.s.Server Started @1059886ms
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=63867}
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45454}
[junit4] 2> 1055999 ERROR (jetty-launcher-4197-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1055999 INFO (jetty-launcher-4197-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-06T02:30:37.810Z
[junit4] 2> 1055999 ERROR (jetty-launcher-4197-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1056000 INFO (jetty-launcher-4197-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 1056000 INFO (jetty-launcher-4197-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1056000 INFO (jetty-launcher-4197-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1056000 INFO (jetty-launcher-4197-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-06T02:30:37.811Z
[junit4] 2> 1056002 INFO (zkConnectionManagerCallback-4202-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1056003 INFO (zkConnectionManagerCallback-4204-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1056003 INFO (jetty-launcher-4197-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1056004 INFO (jetty-launcher-4197-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1056009 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160c94e43840002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1056009 INFO (jetty-launcher-4197-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41494/solr
[junit4] 2> 1056010 INFO (jetty-launcher-4197-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41494/solr
[junit4] 2> 1056012 INFO (zkConnectionManagerCallback-4208-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1056013 INFO (zkConnectionManagerCallback-4212-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1056015 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1056016 INFO (zkConnectionManagerCallback-4216-thread-1-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1056148 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1056150 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45454_solr
[junit4] 2> 1056152 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.Overseer Overseer (id=99300529779507206-127.0.0.1:45454_solr-n_0000000000) starting
[junit4] 2> 1056162 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45454_solr
[junit4] 2> 1056167 INFO (OverseerStateUpdate-99300529779507206-127.0.0.1:45454_solr-n_0000000000) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1056171 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1056186 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1056188 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63867_solr
[junit4] 2> 1056189 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1056193 INFO (zkCallback-4215-thread-2-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1056247 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 1056253 WARN (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 1056257 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 1056259 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 1056259 WARN (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 1056261 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 1056327 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 1056327 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 1056345 INFO (jetty-launcher-4197-thread-2-EventThread) [n:127.0.0.1:45454_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 1056346 INFO (jetty-launcher-4197-thread-1-EventThread) [n:127.0.0.1:63867_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 1056439 WARN (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 1056445 WARN (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 1056481 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45454.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@298f6d8c
[junit4] 2> 1056493 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45454.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@298f6d8c
[junit4] 2> 1056493 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45454.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@298f6d8c
[junit4] 2> 1056494 INFO (jetty-launcher-4197-thread-2) [n:127.0.0.1:45454_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_A5CD4223B6915CDB-001/tempDir-001/node2/.
[junit4] 2> 1056496 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_63867.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@298f6d8c
[junit4] 2> 1056502 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_63867.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@298f6d8c
[junit4] 2> 1056502 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_63867.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@298f6d8c
[junit4] 2> 1056502 INFO (jetty-launcher-4197-thread-1) [n:127.0.0.1:63867_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_A5CD4223B6915CDB-001/tempDir-001/node1/.
[junit4] 2> 1056524 INFO (zkConnectionManagerCallback-4222-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1056543 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
[junit4] 2> 1056557 INFO (qtp1228116405-11185) [ ] o.a.s.s.SolrDispatchFilter Error authenticating
[junit4] 2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
[junit4] 2> at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1056557 WARN (qtp1228116405-11185) [ ] o.e.j.s.ServletHandler /solr/admin/collections
[junit4] 2> org.apache.solr.common.SolrException: Error during request authentication,
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
[junit4] 2> at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> ... 26 more
[junit4] 2> 1056558 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
[junit4] 2> 1056574 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
[junit4] 2> 1056580 INFO (qtp1228116405-11189) [n:127.0.0.1:45454_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1056580 INFO (qtp1228116405-11189) [n:127.0.0.1:45454_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1056587 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
[junit4] 2> 1056605 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
[junit4] 2> 1056613 WARN (qtp1228116405-11188) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1072331 WARN (jetty-launcher-4197-thread-1-SendThread(127.0.0.1:41494)) [n:127.0.0.1:63867_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15828ms for sessionid 0x160c94e43840005
[junit4] 2> 1072331 WARN (jetty-launcher-4197-thread-1-SendThread(127.0.0.1:41494)) [n:127.0.0.1:63867_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15845ms for sessionid 0x160c94e43840008
[junit4] 2> 1072332 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160c94e43840008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1072332 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15837ms for sessionid 0x160c94e43840006
[junit4] 2> 1072332 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160c94e43840005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1072332 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15846ms for sessionid 0x160c94e43840007
[junit4] 2> 1072333 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160c94e43840007, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1072361 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160c94e43840006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1072380 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
[junit4] 2> 1072420 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
[junit4] 2> 1072422 WARN (qtp1228116405-11190) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1072433 INFO (jetty-launcher-4197-thread-2-EventThread) [n:127.0.0.1:45454_solr ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
[junit4] 2> 1072435 WARN (zkConnectionManagerCallback-4216-thread-1-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@61e2c79 name: ZooKeeperConnection Watcher:127.0.0.1:41494/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1072436 WARN (zkConnectionManagerCallback-4216-thread-1-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1072436 WARN (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5301082c name: ZooKeeperConnection Watcher:127.0.0.1:41494/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 1072436 WARN (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 1072450 INFO (qtp1228116405-11190) [n:127.0.0.1:45454_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1072451 INFO (qtp1228116405-11190) [n:127.0.0.1:45454_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1072451 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
[junit4] 2> 1072476 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
[junit4] 2> 1072477 WARN (qtp1228116405-11182) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1087533 WARN (jetty-launcher-4197-thread-1-SendThread(127.0.0.1:41494)) [n:127.0.0.1:63867_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x160c94e43840008 has expired
[junit4] 2> 1087533 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x160c94e43840007 has expired
[junit4] 2> 1087533 WARN (jetty-launcher-4197-thread-1-SendThread(127.0.0.1:41494)) [n:127.0.0.1:63867_solr ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x160c94e43840005 has expired
[junit4] 2> 1087535 INFO (zkConnectionManagerCallback-4216-thread-1-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1087537 WARN (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5301082c name: ZooKeeperConnection Watcher:127.0.0.1:41494/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
[junit4] 2> 1087537 WARN (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 1087537 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1087539 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
[junit4] 2> 1087546 WARN (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
[junit4] 2> 1087550 INFO (zkCallback-4215-thread-5-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 1087561 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr-EventThread) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1087561 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
[junit4] 2> 1087562 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
[junit4] 2> 1087588 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidRemote
[junit4] 2> 1087589 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 1087589 WARN (qtp1228116405-11185) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1087595 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidRemote
[junit4] 2> 1087611 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63867_solr
[junit4] 2> 1087622 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1087622 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
[junit4] 2> 1087622 INFO (zkConnectionManagerCallback-4214-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:true
[junit4] 2> 1087622 INFO (zkCallback-4215-thread-5-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1087632 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
[junit4] 2> 1087633 WARN (qtp1228116405-11183) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1087636 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
[junit4] 2> 1087668 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
[junit4] 2> 1087668 WARN (qtp1228116405-11187) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1087671 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
[junit4] 2> 1087709 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
[junit4] 2> 1087709 WARN (qtp1228116405-11189) [n:127.0.0.1:45454_solr ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1087710 INFO (qtp1228116405-11189) [n:127.0.0.1:45454_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1087710 INFO (qtp1228116405-11189) [n:127.0.0.1:45454_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1087710 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
[junit4] 2> 1087746 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
[junit4] 2> 1087746 WARN (qtp1228116405-11188) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1087751 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
[junit4] 2> 1087782 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
[junit4] 2> 1087783 WARN (qtp1228116405-11188) [ ] o.a.h.s.a.s.AuthenticationFilter AuthenticationToken ignored: Invalid AuthenticationToken type
[junit4] 2> 1087800 WARN (qtp1228116405-11188) [ ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
[junit4] 2> id: invalid user name: "bar"
[junit4] 2>
[junit4] 2> 1087803 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
[junit4] 2> 1087833 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testForwarding
[junit4] 2> 1087835 INFO (zkConnectionManagerCallback-4225-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1088374 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x160c94e4384000b, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1088376 INFO (zkConnectionManagerCallback-4228-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1088378 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[A5CD4223B6915CDB]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1088379 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[A5CD4223B6915CDB]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41494/solr ready
[junit4] 2> 1088381 INFO (qtp1228116405-11185) [n:127.0.0.1:45454_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1088384 INFO (OverseerThreadFactory-3488-thread-1-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.CreateCollectionCmd Create collection forwardingCollection
[junit4] 2> 1088504 INFO (OverseerStateUpdate-99300529779507206-127.0.0.1:45454_solr-n_0000000000) [n:127.0.0.1:45454_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"forwardingCollection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"forwardingCollection_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:63867/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1088719 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1088830 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1088830 INFO (zkCallback-4213-thread-7-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1089756 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 1089777 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema [forwardingCollection_shard1_replica_n1] Schema name=test
[junit4] 2> 1089846 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1089867 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'forwardingCollection_shard1_replica_n1' using configuration from collection forwardingCollection, trusted=true
[junit4] 2> 1089867 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_63867.solr.core.forwardingCollection.shard1.replica_n1' (registry 'solr.core.forwardingCollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@298f6d8c
[junit4] 2> 1089867 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1089867 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [[forwardingCollection_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_A5CD4223B6915CDB-001/tempDir-001/node1/forwardingCollection_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_A5CD4223B6915CDB-001/tempDir-001/node1/./forwardingCollection_shard1_replica_n1/data/]
[junit4] 2> 1089869 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=93.0849609375, floorSegmentMB=1.3486328125, forceMergeDeletesPctAllowed=15.201932182208239, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5503005518125337
[junit4] 2> 1089914 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1089914 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1089915 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=93.0224609375, floorSegmentMB=1.7216796875, forceMergeDeletesPctAllowed=22.681379877343318, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1089915 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@71d69b49[forwardingCollection_shard1_replica_n1] main]
[junit4] 2> 1089916 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
[junit4] 2> 1089919 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 1089932 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1089932 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
[junit4] 2> 1089933 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
[junit4] 2> 1089933 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
[junit4] 2> 1089933 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
[junit4] 2> 1089933 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
[junit4] 2> 1089933 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
[junit4] 2> 1089934 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
[junit4] 2> 1089934 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
[junit4] 2> 1089935 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
[junit4] 2> 1089936 INFO (searcherExecutor-3493-thread-1-processing-n:127.0.0.1:63867_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] Registered new searcher Searcher@71d69b49[forwardingCollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1089937 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1089937 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1089937 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:63867/solr/forwardingCollection_shard1_replica_n1/
[junit4] 2> 1089937 ERROR (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy No UpdateLog found - cannot sync
[junit4] 2> 1089937 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1089937 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1089942 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:63867/solr/forwardingCollection_shard1_replica_n1/ shard1
[junit4] 2> 1090046 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1090046 INFO (zkCallback-4213-thread-7-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1090095 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1090099 INFO (qtp1583179464-11195) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1380
[junit4] 2> 1090106 INFO (qtp1228116405-11185) [n:127.0.0.1:45454_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1090202 INFO (zkCallback-4213-thread-7-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1090202 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1090390 INFO (OverseerCollectionConfigSetProcessor-99300529779507206-127.0.0.1:45454_solr-n_0000000000) [n:127.0.0.1:45454_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1091112 INFO (qtp1228116405-11185) [n:127.0.0.1:45454_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=2732
[junit4] 2> 1091114 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[A5CD4223B6915CDB]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: forwardingCollection failOnTimeout:true timeout (sec):100
[junit4] 2> 1091115 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[A5CD4223B6915CDB]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: forwardingCollection
[junit4] 2> 1091119 INFO (qtp1228116405-11187) [n:127.0.0.1:45454_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1091125 INFO (qtp1583179464-11193) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 1091132 INFO (qtp1583179464-11197) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 1091133 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[A5CD4223B6915CDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testForwarding
[junit4] 2> 1091139 INFO (jetty-closer-4198-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@34617391{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 1091139 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=584192851
[junit4] 2> 1091140 INFO (jetty-closer-4198-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1c2c5c1{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 1091140 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=228343229
[junit4] 2> 1091140 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 1091140 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@64af3736: rootName = solr_45454, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@23e08821
[junit4] 2> 1091140 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 1091140 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3dc6d949: rootName = solr_63867, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@745a9323
[junit4] 2> 1091149 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 1091149 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@532f8994: rootName = solr_45454, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7b07e07
[junit4] 2> 1091149 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 1091150 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5c04023: rootName = solr_63867, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7b07e07
[junit4] 2> 1091167 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 1091167 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f93bf1e: rootName = solr_45454, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@68c82c79
[junit4] 2> 1091167 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 1091167 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@71161d56: rootName = solr_63867, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@68c82c79
[junit4] 2> 1091170 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:45454_solr
[junit4] 2> 1091171 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:63867_solr
[junit4] 2> 1091172 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 1091173 INFO (zkCallback-4215-thread-5-processing-n:127.0.0.1:45454_solr) [n:127.0.0.1:45454_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
[junit4] 2> 1091175 INFO (jetty-closer-4198-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 1091175 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
[junit4] 2> 1091177 INFO (coreCloseExecutor-3498-thread-1) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@2f0cbd98
[junit4] 2> 1091177 INFO (coreCloseExecutor-3498-thread-1) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.forwardingCollection.shard1.replica_n1, tag=789364120
[junit4] 2> 1091177 INFO (coreCloseExecutor-3498-thread-1) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3455823b: rootName = solr_63867, domain = solr.core.forwardingCollection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.forwardingCollection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@51835682
[junit4] 2> 1091179 INFO (jetty-closer-4198-thread-1) [ ] o.a.s.c.Overseer Overseer (id=99300529779507206-127.0.0.1:45454_solr-n_0000000000) closing
[junit4] 2> 1091179 INFO (OverseerStateUpdate-99300529779507206-127.0.0.1:45454_solr-n_0000000000) [n:127.0.0.1:45454_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:45454_solr
[junit4] 2> 1091180 WARN (OverseerAutoScalingTriggerThread-99300529779507206-127.0.0.1:45454_solr-n_0000000000) [n:127.0.0.1:45454_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1091186 INFO (zkCallback-4213-thread-1-processing-n:127.0.0.1:63867_solr) [n:127.0.0.1:63867_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:63867_solr
[junit4] 2> 1091189 ERROR (Thread[Thread-2074,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:45454_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 1091195 INFO (coreCloseExecutor-3498-thread-1) [n:127.0.0.1:63867_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.forwardingCollection.shard1.leader, tag=789364120
[junit4] 2> 1091200 INFO (jetty-closer-4198-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@b435cd4{/solr,null,UNAVAILABLE}
[junit4] 2> 1091203 ERROR (Thread[Thread-2076,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:63867_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 1091207 INFO (jetty-closer-4198-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@604ca546{/solr,null,UNAVAILABLE}
[junit4] 2> 1091209 ERROR (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1091209 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41494 41494
[junit4] 2> 1102934 ERROR (jetty-launcher-4197-thread-2-EventThread) [n:127.0.0.1:45454_solr ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
[junit4] 2> java.lang.IllegalStateException: Client is not started
[junit4] 2> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
[junit4] 2> at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:477)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceFacade.getZooKeeper(NamespaceFacade.java:112)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> 1102935 INFO (Thread-2057) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41494 41494
[junit4] 2> 1102937 WARN (Thread-2057) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2> 4 /solr/clusterprops.json
[junit4] 2> 3 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/collections/forwardingCollection/state.json
[junit4] 2> 4 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2> 2 /solr/overseer_elect/election/99300529779507206-127.0.0.1:45454_solr-n_0000000000
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/live_nodes
[junit4] 2> 4 /solr/collections
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
[junit4] 2>
[junit4] 2> 1102939 WARN (jetty-launcher-4197-thread-2-EventThread) [n:127.0.0.1:45454_solr ] o.a.c.ConnectionState Session expired event received
[junit4] 2> 1102940 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1102982 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[A5CD4223B6915CDB]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Jan 06, 2018 2:31:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
[junit4] 2> 1104042 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1105146 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1106248 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1107354 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1108456 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1109560 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1110664 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1111765 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1112868 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1113970 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1115071 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1116178 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1117280 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1118386 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1119491 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1120593 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1121703 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1122804 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(N
[...truncated too long message...]
1141)
[junit4] 2> 1125251 WARN (jetty-launcher-4197-thread-1-SendThread(127.0.0.1:41494)) [n:127.0.0.1:63867_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> 1125558 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.c.StaticHostProvider Unexpected exception
[junit4] 2> java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] 2> 1125559 WARN (jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494)) [n:127.0.0.1:45454_solr ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
[junit4] 2> java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> Jan 06, 2018 2:31:47 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=11305, name=jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=en-NZ, timezone=Asia/Samarkand
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=4,free=149570448,total=340500480
[junit4] 2> NOTE: All tests run in this JVM: [TestMultiValuedNumericRangeQuery, OverseerStatusTest, BlockJoinFacetRandomTest, TestManagedSchemaThreadSafety, DistributedQueryComponentCustomSortTest, SmileWriterTest, DistributedFacetSimpleRefinementLongTailTest, TestNodeAddedTrigger, BufferStoreTest, TestSolrCoreSnapshots, ShufflingReplicaListTransformerTest, SolrCLIZkUtilsTest, TlogReplayBufferedWhileIndexingTest, TestJavabinTupleStreamParser, TestSubQueryTransformerCrossCore, DistanceFunctionTest, ImplicitSnitchTest, SolrXmlInZkTest, TestBinaryField, TestInitParams, CdcrReplicationDistributedZkTest, UniqFieldsUpdateProcessorFactoryTest, TestHttpShardHandlerFactory, TestSchemaManager, TestSQLHandler, SyncSliceTest, CursorMarkTest, SearchRateTriggerTest, SolrRequestParserTest, StatsReloadRaceTest, RecoveryZkTest, DistributedFacetPivotSmallTest, OverseerModifyCollectionTest, RulesTest, StatsComponentTest, TestPHPSerializedResponseWriter, TestSurroundQueryParser, TestRetrieveFieldsOptimizer, TestReloadDeadlock, TimeZoneUtilsTest, AutoAddReplicasIntegrationTest, LeaderInitiatedRecoveryOnShardRestartTest, TestScoreJoinQPScore, CoreMergeIndexesAdminHandlerTest, HLLSerializationTest, DistributedSpellCheckComponentTest, ClusterStateTest, UnloadDistributedZkTest, TestConfigSets, ZkControllerTest, SolrCloudReportersTest, HdfsUnloadDistributedZkTest, TestNestedDocsSort, TestXmlQParser, SortByFunctionTest, SearchHandlerTest, TestFreeTextSuggestions, TestDistributedSearch, TestPolicyCloud, TestSlowCompositeReaderWrapper, MoveReplicaHDFSFailoverTest, DistributedSuggestComponentTest, CdcrBidirectionalTest, TestGenericDistributedQueue, TestDeleteCollectionOnDownNodes, ClassificationUpdateProcessorFactoryTest, TestSimpleTextCodec, TestUpdate, ExecutePlanActionTest, HttpTriggerListenerTest, EnumFieldTest, TestClassicSimilarityFactory, TestRandomCollapseQParserPlugin, TestOverriddenPrefixQueryForCustomFieldType, SortSpecParsingTest, TestPhraseSuggestions, PeerSyncWithIndexFingerprintCachingTest, TestCopyFieldCollectionResource, InfixSuggestersTest, AtomicUpdateProcessorFactoryTest, ZkStateReaderTest, RAMDirectoryFactoryTest, RequiredFieldsTest, TestManagedSynonymFilterFactory, TestImplicitCoreProperties, TestSolrConfigHandlerCloud, BasicAuthIntegrationTest, TestPrepRecovery, SolrJmxReporterTest, SolrMetricManagerTest, AutoScalingHandlerTest, SpellCheckCollatorTest, DistributedFacetExistsSmallTest, TestSolrCloudWithSecureImpersonation]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=A5CD4223B6915CDB -Dtests.slow=true -Dtests.locale=en-NZ -Dtests.timezone=Asia/Samarkand -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | TestSolrCloudWithSecureImpersonation (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
[junit4] > 1) Thread[id=11305, name=jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] > 2) Thread[id=11306, name=jetty-launcher-4197-thread-2-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
[junit4] > 3) Thread[id=11235, name=jetty-launcher-4197-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] > at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] > at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] > at __randomizedtesting.SeedInfo.seed([A5CD4223B6915CDB]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=11305, name=jetty-launcher-4197-thread-2-SendThread(127.0.0.1:41494), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] > at __randomizedtesting.SeedInfo.seed([A5CD4223B6915CDB]:0)
[junit4] Completed [216/765 (1!)] on J0 in 70.21s, 12 tests, 2 errors <<< FAILURES!
[...truncated 49889 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 375 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/375/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 1) Thread[id=37645, name=jetty-launcher-8147-thread-2-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 2) Thread[id=37644, name=jetty-launcher-8147-thread-2-SendThread(127.0.0.1:59478), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
1) Thread[id=37645, name=jetty-launcher-8147-thread-2-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
2) Thread[id=37644, name=jetty-launcher-8147-thread-2-SendThread(127.0.0.1:59478), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
at __randomizedtesting.SeedInfo.seed([A5B9F22229685486]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=37644, name=jetty-launcher-8147-thread-2-SendThread(127.0.0.1:59478), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=37644, name=jetty-launcher-8147-thread-2-SendThread(127.0.0.1:59478), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
at __randomizedtesting.SeedInfo.seed([A5B9F22229685486]:0)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([A5B9F22229685486:8E462779B3104156]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState(TestTriggerIntegration.java:307)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12063 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_A5B9F22229685486-001/init-core-data-001
[junit4] 2> 714550 WARN (SUITE-TestTriggerIntegration-seed#[A5B9F22229685486]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=3 numCloses=3
[junit4] 2> 714553 INFO (SUITE-TestTriggerIntegration-seed#[A5B9F22229685486]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 714555 INFO (SUITE-TestTriggerIntegration-seed#[A5B9F22229685486]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 714555 INFO (SUITE-TestTriggerIntegration-seed#[A5B9F22229685486]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 714645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 714645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 714645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 714645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 714645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 714653 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 714653 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
[junit4] 2> 714657 DEBUG (ScheduledTrigger-1947-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714714 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
[junit4] 2> 714718 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 714718 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 714722 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 714722 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 714722 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 714732 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 714733 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 714734 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 714734 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 714734 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 714734 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 714734 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 714734 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 714734 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 714734 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 714734 DEBUG (ScheduledTrigger-1950-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714754 DEBUG (ScheduledTrigger-1950-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714775 DEBUG (ScheduledTrigger-1950-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714797 DEBUG (ScheduledTrigger-1950-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714819 DEBUG (ScheduledTrigger-1950-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714835 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 1
[junit4] 2> 714835 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 714836 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 714837 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 714837 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 714838 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 714838 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 714838 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 714838 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10001_solr
[junit4] 2> 714838 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 714838 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 714838 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714838 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
[junit4] 2> 714858 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714879 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714900 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714920 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714948 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714948 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD TRIGGERS
[junit4] 2> 714968 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714968 DEBUG (simCloudManagerPool-1952-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 714969 DEBUG (simCloudManagerPool-1952-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 714969 DEBUG (simCloudManagerPool-1952-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 714969 DEBUG (simCloudManagerPool-1952-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 714969 DEBUG (simCloudManagerPool-1952-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 714969 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 714970 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 714986 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 714988 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 714988 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 714989 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 714987 DEBUG (simCloudManagerPool-1952-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 714989 DEBUG (simCloudManagerPool-1952-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 714989 DEBUG (simCloudManagerPool-1952-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 714989 DEBUG (simCloudManagerPool-1952-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 714989 DEBUG (simCloudManagerPool-1952-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10000_solr, 127.0.0.1:10002_solr]
[junit4] 2> 714989 DEBUG (simCloudManagerPool-1952-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 714989 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD NODE 1
[junit4] 2> 714990 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 714990 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 714990 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 714990 DEBUG (ScheduledTrigger-1953-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 715009 DEBUG (ScheduledTrigger-1953-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715011 DEBUG (ScheduledTrigger-1953-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 715011 DEBUG (ScheduledTrigger-1953-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715011 DEBUG (ScheduledTrigger-1953-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715011 DEBUG (ScheduledTrigger-1953-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10003_solr at time 2904833402273924
[junit4] 2> 715032 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715033 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 715033 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715033 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715033 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10003_solr] added at times [2904833402273924], now=2904834511428874
[junit4] 2> 715033 DEBUG (ScheduledTrigger-1953-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a51ee3b130084Tahighjtruulsnbyv23v5sfeb0",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":2904833402273924,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904833402273924],
[junit4] 2> "nodeNames":["127.0.0.1:10003_solr"]}}
[junit4] 2> 715036 DEBUG (AutoscalingActionExecutor-1954-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a51ee3b130084Tahighjtruulsnbyv23v5sfeb0",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":2904833402273924,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904833402273924],
[junit4] 2> "_enqueue_time_":2904834567956524,
[junit4] 2> "nodeNames":["127.0.0.1:10003_solr"]}}
[junit4] 2> 715039 DEBUG (AutoscalingActionExecutor-1954-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 112 ms for event id=a51ee3b130084Tahighjtruulsnbyv23v5sfeb0
[junit4] 2> 715059 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715059 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 715059 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715059 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715079 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715079 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 715079 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715079 DEBUG (ScheduledTrigger-1953-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715090 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 2
[junit4] 2> 715090 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 715091 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 715092 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 715093 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 715093 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10000_solr
[junit4] 2> 715094 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 715094 DEBUG (ScheduledTrigger-1956-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 715094 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 715094 DEBUG (ScheduledTrigger-1956-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10000_solr
[junit4] 2> 715094 DEBUG (ScheduledTrigger-1956-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715094 DEBUG (ScheduledTrigger-1956-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 715094 DEBUG (ScheduledTrigger-1956-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715094 DEBUG (ScheduledTrigger-1956-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10000_solr
[junit4] 2> 715115 DEBUG (ScheduledTrigger-1956-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 715121 DEBUG (ScheduledTrigger-1956-thread-2) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10000_solr]
[junit4] 2> 715121 DEBUG (ScheduledTrigger-1956-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a51ef32def11aTahighjtruulsnbyv23v5sfeb3",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2904837559611674,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904837559611674],
[junit4] 2> "nodeNames":["127.0.0.1:10000_solr"]}}
[junit4] 2> 715122 DEBUG (AutoscalingActionExecutor-1957-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a51ef32def11aTahighjtruulsnbyv23v5sfeb3",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2904837559611674,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904837559611674],
[junit4] 2> "_enqueue_time_":2904838916709424,
[junit4] 2> "nodeNames":["127.0.0.1:10000_solr"]}}
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 2
[junit4] 2> 715122 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 2
[junit4] 2> 715123 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10000_solr
[junit4] 2> 715123 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10001_solr
[junit4] 2> 715124 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 715124 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 715124 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 715124 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 715125 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 715125 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 715125 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 3
[junit4] 2> 715125 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 715127 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
[junit4] 2> 715127 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 715128 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 715128 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger
[junit4] 2> 715128 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 715129 DEBUG (ScheduledTrigger-1956-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715129 DEBUG (ScheduledTrigger-1956-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 715133 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
[junit4] 2> 715133 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 715133 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 715133 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10001_solr
[junit4] 2> 715133 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10000_solr
[junit4] 2> 715133 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 715135 DEBUG (AutoscalingActionExecutor-1957-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 647 ms for event id=a51ef32def11aTahighjtruulsnbyv23v5sfeb3
[junit4] 2> 715135 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 715135 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 715135 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 715136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
[junit4] 2> 715136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 715136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 715136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 715136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 715136 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 715136 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 715136 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715157 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715179 DEBUG (ScheduledTrigger-1959-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715199 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715231 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715244 DEBUG (simCloudManagerPool-1958-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 715244 DEBUG (simCloudManagerPool-1958-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 715245 DEBUG (simCloudManagerPool-1958-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715245 DEBUG (simCloudManagerPool-1958-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 715245 DEBUG (simCloudManagerPool-1958-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715245 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 715245 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 715246 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 715246 DEBUG (ScheduledTrigger-1959-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 715246 DEBUG (ScheduledTrigger-1959-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715246 DEBUG (ScheduledTrigger-1959-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10004_solr at time 2904845154404774
[junit4] 2> 715251 DEBUG (ScheduledTrigger-1959-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715266 DEBUG (ScheduledTrigger-1959-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 715270 DEBUG (ScheduledTrigger-1959-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715272 DEBUG (ScheduledTrigger-1959-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715277 DEBUG (simCloudManagerPool-1958-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 715278 DEBUG (simCloudManagerPool-1958-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 715278 DEBUG (simCloudManagerPool-1958-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715278 DEBUG (simCloudManagerPool-1958-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 715278 DEBUG (simCloudManagerPool-1958-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
[junit4] 2> 715278 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 715279 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 715279 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 715280 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 715280 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715280 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing registered processor for nodes: [127.0.0.1:10004_solr] added at times [2904845154404774], now=2904846854584724
[junit4] 2> 715280 DEBUG (ScheduledTrigger-1959-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a51f0f78e49a6Tahighjtruulsnbyv23v5sfeb6",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":2904845154404774,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904845154404774],
[junit4] 2> "nodeNames":["127.0.0.1:10004_solr"]}}
[junit4] 2> 715281 DEBUG (AutoscalingActionExecutor-1960-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a51f0f78e49a6Tahighjtruulsnbyv23v5sfeb6",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":2904845154404774,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904845154404774],
[junit4] 2> "_enqueue_time_":2904846862069324,
[junit4] 2> "nodeNames":["127.0.0.1:10004_solr"]}}
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 3
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 3
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 715281 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 2
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_restore_trigger
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 715282 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 715283 DEBUG (AutoscalingActionExecutor-1960-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 98 ms for event id=a51f0f78e49a6Tahighjtruulsnbyv23v5sfeb6
[junit4] 2> 715287 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
[junit4] 2> 715287 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 715287 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 715288 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 715303 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 715304 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 715304 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 715304 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715304 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
[junit4] 2> 715304 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 715304 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 715304 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
[junit4] 2> 715304 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715325 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715351 DEBUG (ScheduledTrigger-1962-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715375 DEBUG (ScheduledTrigger-1962-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715409 DEBUG (simCloudManagerPool-1961-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 715409 DEBUG (simCloudManagerPool-1961-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 715430 DEBUG (ScheduledTrigger-1962-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715433 DEBUG (simCloudManagerPool-1961-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715433 DEBUG (simCloudManagerPool-1961-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10005_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715433 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 715433 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 715433 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 715433 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
[junit4] 2> 715434 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 715434 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10005_solr
[junit4] 2> 715451 DEBUG (ScheduledTrigger-1962-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715451 DEBUG (ScheduledTrigger-1962-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10005_solr
[junit4] 2> 715506 DEBUG (ScheduledTrigger-1962-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715525 DEBUG (simCloudManagerPool-1961-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 715551 DEBUG (simCloudManagerPool-1961-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 715551 DEBUG (simCloudManagerPool-1961-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715551 DEBUG (simCloudManagerPool-1961-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715551 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
[junit4] 2> 715552 DEBUG (ScheduledTrigger-1962-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 715552 DEBUG (ScheduledTrigger-1962-thread-2) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10005_solr]
[junit4] 2> 715552 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 715552 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10005_solr
[junit4] 2> 715552 DEBUG (ScheduledTrigger-1962-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a51f32918ae42Tahighjtruulsnbyv23v5sfeb9",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":2904854575492674,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904854575492674],
[junit4] 2> "nodeNames":["127.0.0.1:10005_solr"]}}
[junit4] 2> 715552 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 715552 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
[junit4] 2> 715552 ERROR (ScheduledTrigger-1962-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Unexpected exception in NodeLostTrigger
[junit4] 2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger node_lost_restore_trigger has been closed.
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.enqueue(ScheduledTriggers.java:487)
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$add$4(ScheduledTriggers.java:268)
[junit4] 2> at org.apache.solr.cloud.autoscaling.NodeLostTrigger.run(NodeLostTrigger.java:159)
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.run(ScheduledTriggers.java:537)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 715553 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 715553 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10005_solr]
[junit4] 2> 715553 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a51f4888afcb2Tahighjtruulsnbyv23v5sfeba",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":2904860471786674,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904860471786674],
[junit4] 2> "nodeNames":["127.0.0.1:10005_solr"]}}
[junit4] 2> 715554 DEBUG (ScheduledTrigger-1962-thread-1) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger processor for lost nodes: [127.0.0.1:10005_solr] is not ready, will try later
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 2
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10005_solr
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 1
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_restore_trigger
[junit4] 2> 715653 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - IGNORED 1
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestTriggerIntegration -Dtests.method=testNodeLostTriggerRestoreState -Dtests.seed=A5B9F22229685486 -Dtests.slow=true -Dtests.locale=no -Dtests.timezone=AET -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.39s J0 | TestTriggerIntegration.testNodeLostTriggerRestoreState <<<
[junit4] > Throwable #1: java.lang.AssertionError: The trigger did not fire at all
[junit4] > at __randomizedtesting.SeedInfo.seed([A5B9F22229685486:8E462779B3104156]:0)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeLostTriggerRestoreState(TestTriggerIntegration.java:307)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 715680 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
[junit4] 2> 715681 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
[junit4] 2> 715681 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
[junit4] 2> 715681 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10005_solr
[junit4] 2> 715681 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 715681 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 715681 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 715681 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 715682 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 715682 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 715682 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
[junit4] 2> 715682 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715682 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion -1
[junit4] 2> 715682 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
[junit4] 2> 715683 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 715683 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
[junit4] 2> 715683 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715703 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715723 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715744 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715771 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 715787 DEBUG (simCloudManagerPool-1964-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 715788 DEBUG (simCloudManagerPool-1964-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
[junit4] 2> 715788 DEBUG (simCloudManagerPool-1964-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715788 DEBUG (simCloudManagerPool-1964-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10002_solr]
[junit4] 2> 715788 DEBUG (simCloudManagerPool-1964-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 715788 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
[junit4] 2> 715789 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 715789 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
[junit4] 2> 715789 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715789 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715789 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10006_solr at time 2904872331161124
[junit4] 2> 715793 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715820 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715820 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715820 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715843 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715843 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715843 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715865 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715865 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715865 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715893 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715893 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715893 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715913 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715913 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715913 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715933 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715933 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715933 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715955 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715955 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715955 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715975 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 715979 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 715979 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 715996 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 716007 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716007 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 716011 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10006_solr] added at times [2904872331161124], now=2904883425110974
[junit4] 2> 716011 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a51f74b6aae24Tahighjtruulsnbyv23v5sfebc",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":2904872331161124,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904872331161124],
[junit4] 2> "nodeNames":["127.0.0.1:10006_solr"]}}
[junit4] 2> 716012 DEBUG (AutoscalingActionExecutor-1966-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a51f74b6aae24Tahighjtruulsnbyv23v5sfebc",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":2904872331161124,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904872331161124],
[junit4] 2> "_enqueue_time_":2904883437274624,
[junit4] 2> "nodeNames":["127.0.0.1:10006_solr"]}}
[junit4] 2> 716014 DEBUG (AutoscalingActionExecutor-1966-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 80 ms for event id=a51f74b6aae24Tahighjtruulsnbyv23v5sfebc
[junit4] 2> 716016 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716033 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716033 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716034 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10007_solr at time 2904884551424924
[junit4] 2> 716037 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716054 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716054 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716058 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716074 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716074 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716079 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716095 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716095 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716099 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716116 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716116 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716122 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716137 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716137 DEBUG (ScheduledTrigger-1965-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716143 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716165 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716165 DEBUG (ScheduledTrigger-1965-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716165 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716186 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716186 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716186 DEBUG (ScheduledTrigger-1965-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716211 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716211 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716211 DEBUG (ScheduledTrigger-1965-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716213 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 716213 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 716214 DEBUG (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 716223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
[junit4] 2> 716224 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10002_solr, 127.0.0.1:10007_solr]
[junit4] 2> 716224 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10004_solr, 127.0.0.1:10006_solr, 127.0.0.1:10002_solr, 127.0.0.1:10007_solr]
[junit4] 2> 716224 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 716224 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion -1
[junit4] 2> 716224 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
[junit4] 2> 716227 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 716227 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10007_solr
[junit4] 2> 716227 DEBUG (ScheduledTrigger-1968-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 716235 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
[junit4] 2> 716235 DEBUG (ScheduledTrigger-1968-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 716235 DEBUG (ScheduledTrigger-1968-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 716235 DEBUG (ScheduledTrigger-1968-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10007_solr] added at times [2904884551424924], now=2904894638478574
[junit4] 2> 716235 DEBUG (ScheduledTrigger-1968-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a51fa23cd1b9cTahighjtruulsnbyv23v5sfebf",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":2904884551424924,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904884551424924],
[junit4] 2> "nodeNames":["127.0.0.1:10007_solr"]}}
[junit4] 2> 716236 DEBUG (AutoscalingActionExecutor-1969-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a51fa23cd1b9cTahighjtruulsnbyv23v5sfebf",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":2904884551424924,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2904884551424924],
[junit4] 2> "_enqueue_time_":2904894646458224,
[junit4] 2> "nodeNames":["127.0.0.1:10007_solr"]}}
[junit4] 2> 716236 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
[junit4] 2> 716236 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 716236 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 716236 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 716236 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 716236 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 4
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 4
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 1
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 3
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 2
[junit4] 2> 716237 INFO (TEST-TestTriggerIntegration.testEventFromRestoredState-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 716239 DEBUG (AutoscalingActionExecutor-1969-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 147 ms for event id=a51fa23cd1b9cTahighjtruulsnbyv23v5sfebf
[junit4] 2> 716242 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A5B9F22229685486]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCooldown
[junit4] 2> 716242 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
[junit4] 2> 716242 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 716244 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[A5B9F22229685486]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 716245 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 716245 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 17
[junit4] 2> 716245 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10002_solr]
[junit4] 2> 716245 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion -1
[junit4] 2> 716245 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 17
[junit4] 2> 716246 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 716246 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 17, lastZnodeVersion 17
[junit4] 2> 716246 DEBUG (ScheduledTrigger-1971-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 716248 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 716266 DEBUG (ScheduledTrigger-1971-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 716286 DEBUG (ScheduledTrigger-1971-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 716306 DEBUG (ScheduledTrigger-1971-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 716326 DEBUG (ScheduledTrigger-1971-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 716345 DEBUG (simCloudManagerPool-1970-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 716346 DEBUG (simCloudManagerPool-1970-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 18
[junit4] 2> 716352 DEBUG (ScheduledTrigger-1971-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 716352 DEBUG (simCloudManagerPool-1970-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10002_solr]
[junit4] 2> 716352 DEBUG (simCloudManagerPool-1970-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 716352 DEBUG (simCloudManagerPool-1970-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10002_solr]
[junit4] 2> 716352 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 18
[junit4] 2> 716352 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 716352 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion 18
[junit4] 2> 716352 DEBUG (ScheduledTrigger-1971-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Runn
[...truncated too long message...]
onentTest, CloneFieldUpdateProcessorFactoryTest, TestDistributedMissingSort, TestSolrCloudSnapshots, TestAnalyzeInfixSuggestions, LegacyCloudClusterPropTest, TestZkChroot, TestSimpleTrackingShardHandler, SimpleCollectionCreateDeleteTest, TestTolerantUpdateProcessorCloud, HttpSolrCallGetCoreTest, ManagedSchemaRoundRobinCloudTest, TestFilteredDocIdSet, ZkFailoverTest, PropertiesRequestHandlerTest, TestFunctionQuery, TestSchemalessBufferedUpdates, SpellPossibilityIteratorTest, TestMacroExpander, SimpleFacetsTest, PolyFieldTest, TestSuggestSpellingConverter, TestJsonFacetRefinement, ExitableDirectoryReaderTest, TestFieldCollectionResource, EchoParamsTest, MissingSegmentRecoveryTest, FileUtilsTest, TestSubQueryTransformerDistrib, SolrPluginUtilsTest, TermVectorComponentTest, TestRestoreCore, ConcurrentDeleteAndCreateCollectionTest, SolrCoreTest, HdfsNNFailoverTest, TestSolrQueryResponse, PreAnalyzedFieldTest, RecoveryAfterSoftCommitTest, TestPullReplica, HdfsRecoveryZkTest, CurrencyFieldTypeTest, TestLuceneMatchVersion, AssignTest, TestBulkSchemaAPI, OpenExchangeRatesOrgProviderTest, LoggingHandlerTest, BaseCdcrDistributedZkTest, TestIntervalFaceting, CollectionStateFormat2Test, DistributedQueryComponentOptimizationTest, MoveReplicaHDFSTest, TestLeaderElectionWithEmptyReplica, LeaderFailoverAfterPartitionTest, TestSolrDeletionPolicy1, DocValuesMissingTest, TermsComponentTest, TriLevelCompositeIdRoutingTest, TestRawTransformer, TestConfigSetProperties, TestStressLiveNodes, TestAddFieldRealTimeGet, TestShortCircuitedRequests, V2ApiIntegrationTest, HdfsAutoAddReplicasIntegrationTest, CoreAdminRequestStatusTest, ConjunctionSolrSpellCheckerTest, TestNumericTokenStream, TestManagedResourceStorage, CollectionsAPIDistributedZkTest, TestReload, SolrMetricReporterTest, BlockJoinFacetSimpleTest, AnalyticsQueryTest, TestCloudRecovery, TestCollationField, TestClusterProperties, TestManagedSchemaAPI, TestInPlaceUpdatesDistrib, TestStandardQParsers, DocValuesTest, InfoHandlerTest, DirectSolrSpellCheckerTest, TestSortByMinMaxFunction, UpdateLogTest, CopyFieldTest, SolrCloudExampleTest, MoreLikeThisHandlerTest, TestRuleBasedAuthorizationPlugin, TestPostingsSolrHighlighter, TestLuceneIndexBackCompat, TestCloudNestedDocsSort, TestConfigSetsAPIExclusivity, TestSolrCLIRunExample, TestExecutePlanAction, SubstringBytesRefFilterTest, TestHalfAndHalfDocValues, CloudExitableDirectoryReaderTest, TestRTGBase, TestValueSourceCache, BlobRepositoryCloudTest, BadComponentTest, TestHighFrequencyDictionaryFactory, TestFoldingMultitermQuery, UninvertDocValuesMergePolicyTest, ActionThrottleTest, HdfsBasicDistributedZkTest, TimeRoutedAliasUpdateProcessorTest, OpenCloseCoreStressTest, TestExceedMaxTermLength, StatelessScriptUpdateProcessorFactoryTest, TestSolrConfigHandler, TestAuthenticationFramework, UpdateParamsTest, LukeRequestHandlerTest, FieldMutatingUpdateProcessorTest, SimplePostToolTest, CoreSorterTest, LeaderInitiatedRecoveryOnCommitTest, JSONWriterTest, TestCloudInspectUtil, TestConfig, BigEndianAscendingWordDeserializerTest, TestObjectReleaseTracker, TestConfigOverlay, LargeFieldTest, TestExportWriter, TestExtendedDismaxParser, SolrShardReporterTest, DataDrivenBlockJoinTest, SpellingQueryConverterTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestAnalyzedSuggestions, TestSmileRequest, TestGraphMLResponseWriter, TestSolrConfigHandlerConcurrent, CustomHighlightComponentTest, MBeansHandlerTest, DocValuesMultiTest, TestConfigsApi, SolrCoreMetricManagerTest, TestBulkSchemaConcurrent, JsonLoaderTest, TestSubQueryTransformer, TestLRUStatsCache, TriggerIntegrationTest, TestManagedStopFilterFactory, TestUtilizeNode, TestNodeLostTrigger, TestQueryUtils, CdcrBootstrapTest, ConnectionManagerTest, TestRandomFaceting, TestCollectionsAPIViaSolrCloudCluster, TestLegacyFieldReuse, TestPerFieldSimilarityWithDefaultOverride, TestStressCloudBlindAtomicUpdates, TestDocumentBuilder, ZkStateWriterTest, UtilsToolTest, FullSolrCloudDistribCmdsTest, BlockJoinFacetDistribTest, DistributedFacetPivotSmallAdvancedTest, TestDelegationWithHadoopAuth, QueryResultKeyTest, OverseerCollectionConfigSetProcessorTest, AutoscalingHistoryHandlerTest, TestRecoveryHdfs, TestBlobHandler, SchemaVersionSpecificBehaviorTest, DocumentBuilderTest, HdfsSyncSliceTest, OverseerRolesTest, SOLR749Test, TestSolrCloudWithKerberosAlt, TestSearchPerf, TestRemoteStreaming, TestCoreContainer, TestSolr4Spatial, StatsComponentTest, PeerSyncTest, TestIndexSearcher, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SuggesterFSTTest, CoreAdminHandlerTest, TestStressLucene, TestTrie, TestCSVLoader, NoCacheHeaderTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DefaultValueUpdateProcessorTest, SolrInfoBeanTest, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, TestSurroundQueryParser, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, QueryParsingTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, HighlighterConfigTest, AlternateDirectoryTest, TestSolrIndexConfig, SolrIndexConfigTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, SampleTest, TestDocSet, MinimalSchemaTest, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCharFilters, SynonymTokenizerTest, TestXIncludeConfig, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TimeZoneUtilsTest, TestFastOutputStream, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, TestSystemIdResolver, DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestLRUCache, TestUtils, ZkNodePropsTest, SystemInfoHandlerTest, UUIDFieldTest, DistributedMLTComponentTest, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, TestJettySolrRunner, ConnectionReuseTest, AddReplicaTest, AliasIntegrationTest, AssignBackwardCompatibilityTest, CleanupOldIndexTest, CollectionReloadTest, CollectionsAPIAsyncDistributedZkTest, DeleteInactiveReplicaTest, DeleteLastCustomShardedReplicaTest, DeleteShardTest, DeleteStatusTest, DistributedQueueTest, DocValuesNotIndexedTest, ForceLeaderTest, MoveReplicaTest, NodeMutatorTest, OverriddenZkACLAndCredentialsProvidersTest, RemoteQueryErrorTest, ReplaceNodeTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, TestCloudDeleteByQuery, TestCloudPivotFacet, TestCollectionAPI, TestConfigSetsAPI, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, TestLeaderInitiatedRecoveryThread, TestLocalFSCloudBackupRestore, TestLockTree, TestMiniSolrCloudClusterSSL, TestPrepRecovery, TestReplicaProperties, TestRequestStatusCollectionAPI, TestSkipOverseerOperations, TestSolrCloudWithSecureImpersonation]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=A5B9F22229685486 -Dtests.slow=true -Dtests.locale=es-AR -Dtests.timezone=Indian/Reunion -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | TestSolrCloudWithSecureImpersonation (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
[junit4] > 1) Thread[id=37645, name=jetty-launcher-8147-thread-2-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
[junit4] > 2) Thread[id=37644, name=jetty-launcher-8147-thread-2-SendThread(127.0.0.1:59478), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] > at __randomizedtesting.SeedInfo.seed([A5B9F22229685486]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=37644, name=jetty-launcher-8147-thread-2-SendThread(127.0.0.1:59478), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] > at __randomizedtesting.SeedInfo.seed([A5B9F22229685486]:0)
[junit4] Completed [571/765 (2!)] on J1 in 64.27s, 12 tests, 2 errors <<< FAILURES!
[...truncated 48785 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 374 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/374/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test
Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_n1","base_url":"http://127.0.0.1:43438","node_name":"127.0.0.1:43438_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/27)={ "pullReplicas":"0", "replicationFactor":"1", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{ "core_node4":{ "core":"c8n_1x3_lf_shard1_replica_n1", "base_url":"http://127.0.0.1:43438", "node_name":"127.0.0.1:43438_", "state":"active", "type":"NRT", "leader":"true"}, "core_node5":{ "core":"c8n_1x3_lf_shard1_replica_n2", "base_url":"http://127.0.0.1:45981", "node_name":"127.0.0.1:45981_", "state":"down", "type":"NRT"}, "core_node6":{ "state":"down", "base_url":"http://127.0.0.1:33388", "core":"c8n_1x3_lf_shard1_replica_n3", "node_name":"127.0.0.1:33388_", "type":"NRT"}}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"1", "autoAddReplicas":"false", "nrtReplicas":"3", "tlogReplicas":"0"}
Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_n1","base_url":"http://127.0.0.1:43438","node_name":"127.0.0.1:43438_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/27)={
"pullReplicas":"0",
"replicationFactor":"1",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{
"core_node4":{
"core":"c8n_1x3_lf_shard1_replica_n1",
"base_url":"http://127.0.0.1:43438",
"node_name":"127.0.0.1:43438_",
"state":"active",
"type":"NRT",
"leader":"true"},
"core_node5":{
"core":"c8n_1x3_lf_shard1_replica_n2",
"base_url":"http://127.0.0.1:45981",
"node_name":"127.0.0.1:45981_",
"state":"down",
"type":"NRT"},
"core_node6":{
"state":"down",
"base_url":"http://127.0.0.1:33388",
"core":"c8n_1x3_lf_shard1_replica_n3",
"node_name":"127.0.0.1:33388_",
"type":"NRT"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"3",
"tlogReplicas":"0"}
at __randomizedtesting.SeedInfo.seed([49336704300C9DDB:C16758DE9EF0F023]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction.testIntegration
Error Message:
Stack Trace:
java.util.ConcurrentModificationException
at __randomizedtesting.SeedInfo.seed([49336704300C9DDB:F952692815333CFE]:0)
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.tearDown(SimSolrCloudTestCase.java:141)
at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12136 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestExecutePlanAction_49336704300C9DDB-001/init-core-data-001
[junit4] 2> 1085617 WARN (SUITE-TestExecutePlanAction-seed#[49336704300C9DDB]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=27 numCloses=27
[junit4] 2> 1085618 INFO (SUITE-TestExecutePlanAction-seed#[49336704300C9DDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1085619 INFO (SUITE-TestExecutePlanAction-seed#[49336704300C9DDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1085619 INFO (SUITE-TestExecutePlanAction-seed#[49336704300C9DDB]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1085634 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1085635 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 1085635 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1085635 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 1085635 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 1085637 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testExecute
[junit4] 2> 1085637 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 1085639 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1085640 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1085641 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1085641 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1085644 WARN (ScheduledTrigger-2994-thread-1) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger ran but was already closed
[junit4] 2> 1085645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1085645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 1085645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1085645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 1085645 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1085646 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1085646 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1085646 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085671 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085696 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085719 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085739 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085749 DEBUG (simCloudManagerPool-2996-thread-1) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: testExecute
[junit4] 2> 1085759 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085760 DEBUG (simCloudManagerPool-2996-thread-1) [ ] o.a.s.c.Assign Identify nodes using default
[junit4] 2> 1085780 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085797 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction Collection ready after 1278ms
[junit4] 2> 1085800 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085802 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.ExecutePlanAction -- processing event: {
[junit4] 2> "id":"1506cac20d752dc0Tbyt3rr1skkxg5lpts9y8p7wx2",
[junit4] 2> "source":"mock_trigger_name",
[junit4] 2> "eventTime":1515121259455000000,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1515121259455000000],
[junit4] 2> "nodeNames":["127.0.0.1:10000_solr"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@27616652, org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction$1@2a29f8c5]}
[junit4] 2> 1085803 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testExecute&targetNode=127.0.0.1:10001_solr&inPlaceMove=true&replica=core_node2
[junit4] 2> 1085821 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085841 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085861 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085882 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085902 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085905 DEBUG (simCloudManagerPool-2996-thread-4) [ ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node4":{
[junit4] 2> "core":"testExecute_shard1_replica_n3",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testExecute",
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "type":"NRT"}}
[junit4] 2> 1085922 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085943 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085963 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1085983 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086004 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086023 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=OVERSEERSTATUS
[junit4] 2> 1086025 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086025 DEBUG (simCloudManagerPool-2996-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testExecute / shard1: {"core_node1":{
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "core":"testExecute_shard1_replica_n1",
[junit4] 2> "leader":"true",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testExecute"}}
[junit4] 2> 1086028 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction Collection ready after 14ms
[junit4] 2> 1086028 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction -------------_ FINAL STATE --------------
[junit4] 2> 1086028 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Node values: {
[junit4] 2> "127.0.0.1:10001_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":2,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":980,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10001_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10001,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
[junit4] 2> "127.0.0.1:10000_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":0,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":1000,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10000_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10000,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0}}
[junit4] 2> 1086029 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Live nodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1086030 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Collection testExecute state: DocCollection(testExecute//clusterstate.json/3)={
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"1",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"2",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "autoCreated":"true",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "replicas":{
[junit4] 2> "core_node1":{
[junit4] 2> "core":"testExecute_shard1_replica_n1",
[junit4] 2> "leader":"true",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node4":{
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "core":"testExecute_shard1_replica_n3",
[junit4] 2> "state":"active",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "type":"NRT"}},
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active"}}}
[junit4] 2> 1086030 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testExecute
[junit4] 2> 1086030 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1086030 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1086030 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1086030 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1086030 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 1086032 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 1
[junit4] 2> 1086032 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1086032 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: [testExecute]
[junit4] 2> 1086032 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 2
[junit4] 2> 1086032 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1086032 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 2
[junit4] 2> 1086034 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * testExecute 2
[junit4] 2> 1086035 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - active 2
[junit4] 2> 1086035 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1086035 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - CREATE 1
[junit4] 2> 1086036 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - DELETESTATUS 2
[junit4] 2> 1086036 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - MOVEREPLICA 1
[junit4] 2> 1086036 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - OVERSEERSTATUS 1
[junit4] 2> 1086036 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - REQUESTSTATUS 2
[junit4] 2> 1086036 INFO (TEST-TestExecutePlanAction.testExecute-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1086046 DEBUG (ScheduledTrigger-2997-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086049 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testIntegration
[junit4] 2> 1086050 DEBUG (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 1086050 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1086050 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1086051 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1086052 DEBUG (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1086052 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1086052 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1086053 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1086054 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 1086054 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1086054 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
[junit4] 2> 1086054 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1086055 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1086055 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1086055 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086076 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086105 DEBUG (ScheduledTrigger-3000-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086125 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086145 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086166 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086186 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086196 DEBUG (simCloudManagerPool-2999-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1086197 DEBUG (simCloudManagerPool-2999-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 1086197 DEBUG (simCloudManagerPool-2999-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1086197 DEBUG (simCloudManagerPool-2999-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 1086197 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 1086198 DEBUG (simCloudManagerPool-2999-thread-2) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: testIntegration
[junit4] 2> 1086198 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1086198 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 1086198 DEBUG (ScheduledTrigger-3000-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 1086206 DEBUG (ScheduledTrigger-3000-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086208 DEBUG (simCloudManagerPool-2999-thread-2) [ ] o.a.s.c.Assign Identify nodes using default
[junit4] 2> 1086219 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 1086227 DEBUG (ScheduledTrigger-3000-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1086231 DEBUG (simCloudManagerPool-2999-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testIntegration / shard1: {"core_node2":{
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "core":"testIntegration_shard1_replica_n2",
[junit4] 2> "leader":"true",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testIntegration"}}
[junit4] 2> 1086239 DEBUG (ScheduledTrigger-3000-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 1086239 DEBUG (ScheduledTrigger-3000-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
[junit4] 2> 1086247 DEBUG (ScheduledTrigger-3000-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 1086247 DEBUG (ScheduledTrigger-3000-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
[junit4] 2> 1086259 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 1086259 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10001_solr]
[junit4] 2> 1086260 DEBUG (ScheduledTrigger-3000-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a2dde8fa6b36bTbyt3rr1skkxg5lpts9y8p7wx3",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2865183683162987,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2865183683162987],
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}}
[junit4] 2> 1086264 DEBUG (AutoscalingActionExecutor-3001-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a2dde8fa6b36bTbyt3rr1skkxg5lpts9y8p7wx3",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2865183683162987,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2865183683162987],
[junit4] 2> "_enqueue_time_":2865184711798887,
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}}
[junit4] 2> 1086266 DEBUG (AutoscalingActionExecutor-3001-thread-1) [ ] o.a.s.c.a.ComputePlanAction -- processing event: {
[junit4] 2> "id":"a2dde8fa6b36bTbyt3rr1skkxg5lpts9y8p7wx3",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2865183683162987,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2865183683162987],
[junit4] 2> "_enqueue_time_":2865184711798887,
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}} with context properties: {BEFORE_ACTION=[compute_plan]}
[junit4] 2> 1086280 INFO (AutoscalingActionExecutor-3001-thread-1) [ ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testIntegration&targetNode=127.0.0.1:10000_solr&inPlaceMove=true&replica=core_node1
[junit4] 2> 1086281 INFO (AutoscalingActionExecutor-3001-thread-1) [ ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2865185764 sessionWrapper.createTime 2865185443385537, this.sessionWrapper.createTime 2865185443385537
[junit4] 2> 1086281 DEBUG (AutoscalingActionExecutor-3001-thread-1) [ ] o.a.s.c.a.ExecutePlanAction -- processing event: {
[junit4] 2> "id":"a2dde8fa6b36bTbyt3rr1skkxg5lpts9y8p7wx3",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2865183683162987,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2865183683162987],
[junit4] 2> "_enqueue_time_":2865184711798887,
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@55920631], AFTER_ACTION=[compute_plan], BEFORE_ACTION=[compute_plan, execute_plan]}
[junit4] 2> 1086282 DEBUG (AutoscalingActionExecutor-3001-thread-1) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testIntegration&targetNode=127.0.0.1:10000_solr&inPlaceMove=true&replica=core_node1
[junit4] 2> 1086383 DEBUG (simCloudManagerPool-2999-thread-7) [ ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node4":{
[junit4] 2> "core":"testIntegration_shard1_replica_n3",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testIntegration",
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "type":"NRT"}}
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction -------------_ FINAL STATE --------------
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Node values: {
[junit4] 2> "127.0.0.1:10001_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":1,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":970,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10001_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10001,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
[junit4] 2> "127.0.0.1:10000_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":2,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":980,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10000_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10000,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0}}
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Live nodes: [127.0.0.1:10000_solr]
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Collection testIntegration state: DocCollection(testIntegration//clusterstate.json/10)={
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"1",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"2",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "autoCreated":"true",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "replicas":{
[junit4] 2> "core_node2":{
[junit4] 2> "core":"testIntegration_shard1_replica_n2",
[junit4] 2> "leader":"true",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node4":{
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "core":"testIntegration_shard1_replica_n3",
[junit4] 2> "state":"active",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "type":"NRT"}},
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active"}}}
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testIntegration
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 1
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 0
[junit4] 2> 1086494 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10001_solr
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: [testIntegration]
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 2
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 2
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 2
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * testIntegration 2
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - active 2
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - CREATE 1
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - DELETESTATUS 1
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - MOVEREPLICA 1
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - REQUESTSTATUS 1
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 1
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 1
[junit4] 2> 1086495 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[49336704300C9DDB]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1086496 DEBUG (AutoscalingActionExecutor-3001-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 11559 ms for event id=a2dde8fa6b36bTbyt3rr1skkxg5lpts9y8p7wx3
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestExecutePlanAction -Dtests.method=testIntegration -Dtests.seed=49336704300C9DDB -Dtests.slow=true -Dtests.locale=hu -Dtests.timezone=Atlantic/St_Helena -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.46s J1 | TestExecutePlanAction.testIntegration <<<
[junit4] > Throwable #1: java.util.ConcurrentModificationException
[junit4] > at __randomizedtesting.SeedInfo.seed([49336704300C9DDB:F952692815333CFE]:0)
[junit4] > at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
[junit4] > at java.util.ArrayList$Itr.next(ArrayList.java:859)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.tearDown(SimSolrCloudTestCase.java:141)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1086496 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1086501 DEBUG (SUITE-TestExecutePlanAction-seed#[49336704300C9DDB]-worker) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestExecutePlanAction_49336704300C9DDB-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1846, maxMBSortInHeap=6.575937696886248, sim=RandomSimilarity(queryNorm=true): {}, locale=hu, timezone=Atlantic/St_Helena
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=186625488,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [TestStressLiveNodes, TestClusterStateMutator, ChaosMonkeyNothingIsSafeWithPullReplicasTest, SearchHandlerTest, TestReplicationHandlerBackup, BlobRepositoryCloudTest, TestFieldCacheSortRandom, TestSort, HdfsDirectoryTest, TestSolrJ, TermVectorComponentTest, RecoveryAfterSoftCommitTest, CdcrBootstrapTest, DistributedMLTComponentTest, AnalysisAfterCoreReloadTest, TestCorePropertiesReload, TermVectorComponentDistributedTest, HdfsNNFailoverTest, TestFieldCollectionResource, CdcrReplicationHandlerTest, TestCrossCoreJoin, HttpPartitionTest, TestGroupingSearch, SynonymTokenizerTest, TestDelegationWithHadoopAuth, TestExportWriter, LargeFieldTest, DeleteStatusTest, TestCloudSchemaless, DistributedFacetPivotLargeTest, TestSolrConfigHandler, TestPerFieldSimilarity, TestLegacyNumericRangeQueryBuilder, TestPivotHelperCode, CollectionsAPIDistributedZkTest, DeleteReplicaTest, TestPostingsSolrHighlighter, TestIndexSearcher, ReplaceNodeTest, TestIntervalFaceting, TestHashPartitioner, TestUseDocValuesAsStored2, SSLMigrationTest, MultiThreadedOCPTest, TestQueryWrapperFilter, TestRandomFaceting, CustomHighlightComponentTest, TestConfigsApi, TestCloudPivotFacet, UpdateLogTest, ConnectionManagerTest, CacheHeaderTest, TestSerializedLuceneMatchVersion, TestCustomSort, TestFaceting, TestSystemIdResolver, AliasIntegrationTest, TestSolrIndexConfig, TestSolrCloudWithKerberosAlt, TestSQLHandlerNonCloud, TestApiFramework, DocValuesTest, TestHalfAndHalfDocValues, SimplePostToolTest, BasicAuthStandaloneTest, TestExceedMaxTermLength, TestMinMaxOnMultiValuedField, TestLockTree, TestCollectionsAPIViaSolrCloudCluster, TestSchemalessBufferedUpdates, TestPerFieldSimilarityWithDefaultOverride, QueryResultKeyTest, TestLRUStatsCache, TestExecutePlanAction]
[junit4] Completed [157/765 (1!)] on J1 in 0.90s, 2 tests, 1 error <<< FAILURES!
[...truncated 667 lines...]
[junit4] Suite: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/init-core-data-001
[junit4] 2> 1922226 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[49336704300C9DDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1922227 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[49336704300C9DDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 1922227 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[49336704300C9DDB]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1922227 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[49336704300C9DDB]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 1922229 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1922230 INFO (Thread-5163) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1922230 INFO (Thread-5163) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1922231 ERROR (Thread-5163) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1922330 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.ZkTestServer start zk server on port:35343
[junit4] 2> 1922332 INFO (zkConnectionManagerCallback-4115-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1922336 INFO (zkConnectionManagerCallback-4117-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1922343 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1922345 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1922347 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1922349 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1922350 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1922352 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1922354 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1922356 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1922357 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1922360 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1922361 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1922363 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 1922517 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1922518 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@238d065c{/,null,AVAILABLE}
[junit4] 2> 1922519 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2b5ee875{HTTP/1.1,[http/1.1]}{127.0.0.1:43335}
[junit4] 2> 1922519 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.Server Started @1927144ms
[junit4] 2> 1922519 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/tempDir-001/control/data, hostContext=/, hostPort=33388, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/control-001/cores}
[junit4] 2> 1922519 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1922519 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 1922519 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1922519 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1922519 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-05T03:14:56.173Z
[junit4] 2> 1922521 INFO (zkConnectionManagerCallback-4119-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1922522 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1922522 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/control-001/solr.xml
[junit4] 2> 1922524 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1922524 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1922525 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1922527 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35343/solr
[junit4] 2> 1922528 INFO (zkConnectionManagerCallback-4123-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1922530 INFO (zkConnectionManagerCallback-4125-thread-1-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1922647 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1922648 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33388_
[junit4] 2> 1922649 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.c.Overseer Overseer (id=99295041675919364-127.0.0.1:33388_-n_0000000000) starting
[junit4] 2> 1922659 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33388_
[junit4] 2> 1922661 INFO (OverseerStateUpdate-99295041675919364-127.0.0.1:33388_-n_0000000000) [n:127.0.0.1:33388_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1922727 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1922733 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1922733 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1922734 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:33388_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/control-001/cores
[junit4] 2> 1922748 INFO (zkConnectionManagerCallback-4131-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1922750 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1922751 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35343/solr ready
[junit4] 2> 1922751 INFO (SocketProxy-Acceptor-33388) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53116,localport=33388], receiveBufferSize:130880
[junit4] 2> 1922752 INFO (SocketProxy-Acceptor-33388) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43335,localport=49967], receiveBufferSize=139264
[junit4] 2> 1922753 INFO (qtp888731167-19412) [n:127.0.0.1:33388_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33388_&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1922755 INFO (OverseerThreadFactory-6412-thread-1-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 1922866 INFO (SocketProxy-Acceptor-33388) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=53900,localport=33388], receiveBufferSize:130880
[junit4] 2> 1922867 INFO (SocketProxy-Acceptor-33388) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43335,localport=37463], receiveBufferSize=139264
[junit4] 2> 1922868 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1922869 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1922983 INFO (zkCallback-4124-thread-1-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1923891 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 1923904 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 1924022 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1924051 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 1924052 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1924052 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1924052 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 1924054 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=3.529296875, floorSegmentMB=0.7939453125, forceMergeDeletesPctAllowed=2.0415430340164886, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1924058 WARN (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1924099 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1924099 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1924100 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1924100 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1924101 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2071206701, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1924101 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1362536c[control_collection_shard1_replica_n1] main]
[junit4] 2> 1924102 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1924103 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1924103 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1924104 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1588720668778692608
[junit4] 2> 1924104 INFO (searcherExecutor-6415-thread-1-processing-n:127.0.0.1:33388_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@1362536c[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1924110 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1924110 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1924110 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33388/control_collection_shard1_replica_n1/
[junit4] 2> 1924110 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1924110 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33388/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 1924111 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1924120 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33388/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 1924221 INFO (zkCallback-4124-thread-2-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1924272 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1924275 INFO (qtp888731167-19414) [n:127.0.0.1:33388_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1407
[junit4] 2> 1924279 INFO (qtp888731167-19412) [n:127.0.0.1:33388_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1924379 INFO (zkCallback-4124-thread-2-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1924758 INFO (OverseerCollectionConfigSetProcessor-99295041675919364-127.0.0.1:33388_-n_0000000000) [n:127.0.0.1:33388_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1925282 INFO (qtp888731167-19412) [n:127.0.0.1:33388_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33388_&wt=javabin&version=2} status=0 QTime=2530
[junit4] 2> 1925286 INFO (zkConnectionManagerCallback-4135-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1925288 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1925289 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35343/solr ready
[junit4] 2> 1925289 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1925290 INFO (SocketProxy-Acceptor-33388) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57696,localport=33388], receiveBufferSize:130880
[junit4] 2> 1925290 INFO (SocketProxy-Acceptor-33388) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43335,localport=64748], receiveBufferSize=139264
[junit4] 2> 1925291 INFO (qtp888731167-19416) [n:127.0.0.1:33388_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1925294 INFO (OverseerThreadFactory-6412-thread-2-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 1925295 WARN (OverseerThreadFactory-6412-thread-2-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 1925500 INFO (qtp888731167-19416) [n:127.0.0.1:33388_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1925500 INFO (qtp888731167-19416) [n:127.0.0.1:33388_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=210
[junit4] 2> 1925615 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/shard-1-001 of type NRT
[junit4] 2> 1925616 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1925617 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5946f080{/,null,AVAILABLE}
[junit4] 2> 1925617 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2bd05523{HTTP/1.1,[http/1.1]}{127.0.0.1:38517}
[junit4] 2> 1925617 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.e.j.s.Server Started @1930243ms
[junit4] 2> 1925617 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=32976, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/shard-1-001/cores}
[junit4] 2> 1925618 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1925618 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.3.0
[junit4] 2> 1925618 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1925618 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1925618 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-05T03:14:59.272Z
[junit4] 2> 1925619 INFO (zkConnectionManagerCallback-4137-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1925621 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1925621 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/shard-1-001/solr.xml
[junit4] 2> 1925623 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1925623 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1925624 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1925626 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35343/solr
[junit4] 2> 1925627 INFO (zkConnectionManagerCallback-4141-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1925630 INFO (zkConnectionManagerCallback-4143-thread-1-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1925640 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:32976_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1925644 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:32976_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1925647 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:32976_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32976_
[junit4] 2> 1925648 INFO (zkCallback-4124-thread-2-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1925648 INFO (zkCallback-4134-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1925649 INFO (zkCallback-4142-thread-1-processing-n:127.0.0.1:32976_) [n:127.0.0.1:32976_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1925716 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:32976_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1925724 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:32976_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1925724 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:32976_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1925725 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [n:127.0.0.1:32976_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/shard-1-001/cores
[junit4] 2> 1925744 INFO (SocketProxy-Acceptor-32976) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=59956,localport=32976], receiveBufferSize:130880
[junit4] 2> 1925745 INFO (SocketProxy-Acceptor-32976) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38517,localport=39068], receiveBufferSize=139264
[junit4] 2> 1925746 INFO (qtp1994004652-19468) [n:127.0.0.1:32976_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:32976_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1925749 INFO (OverseerCollectionConfigSetProcessor-99295041675919364-127.0.0.1:33388_-n_0000000000) [n:127.0.0.1:33388_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1925750 INFO (OverseerThreadFactory-6412-thread-3-processing-n:127.0.0.1:33388_) [n:127.0.0.1:33388_ ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:32976_ for creating new replica
[junit4] 2> 1925752 INFO (SocketProxy-Acceptor-32976) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52607,localport=32976], receiveBufferSize:130880
[junit4] 2> 1925753 INFO (SocketProxy-Acceptor-32976) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38517,localport=54326], receiveBufferSize=139264
[junit4] 2> 1925753 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1925754 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1926770 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
[junit4] 2> 1926796 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
[junit4] 2> 1926885 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 1926910 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
[junit4] 2> 1926911 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f36a3f5
[junit4] 2> 1926911 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1926911 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
[junit4] 2> 1926912 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=3.529296875, floorSegmentMB=0.7939453125, forceMergeDeletesPctAllowed=2.0415430340164886, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1926914 WARN (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1926962 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1926962 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1926963 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1926963 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1926964 INFO (qtp1994004652-19470) [n:127.0.0.1:32976_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMer
[...truncated too long message...]
2> 2051372 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2051379 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35343 35343
[junit4] 2> 2061434 INFO (Thread-5163) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35343 35343
[junit4] 2> 2061435 WARN (Thread-5163) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2> 4 /solr/configs/conf1
[junit4] 2> 3 /solr/collections/c8n_1x3_lf/state.json
[junit4] 2> 3 /solr/collections/collection1/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 3 /solr/autoscaling.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/live_nodes
[junit4] 2> 6 /solr/collections
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> 2061435 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SocketProxy Closing 6 connections to: http://127.0.0.1:32976/, target: http://127.0.0.1:38517/
[junit4] 2> 2061435 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SocketProxy Closing 7 connections to: http://127.0.0.1:43438/, target: http://127.0.0.1:60060/
[junit4] 2> 2061435 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SocketProxy Closing 0 connections to: http://127.0.0.1:45981/, target: http://127.0.0.1:59416/
[junit4] 2> 2061436 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[49336704300C9DDB]) [ ] o.a.s.c.SocketProxy Closing 1 connections to: http://127.0.0.1:33388/, target: http://127.0.0.1:43335/
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=LeaderFailoverAfterPartitionTest -Dtests.method=test -Dtests.seed=49336704300C9DDB -Dtests.slow=true -Dtests.locale=ar -Dtests.timezone=America/Moncton -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 139s J1 | LeaderFailoverAfterPartitionTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node4:{"core":"c8n_1x3_lf_shard1_replica_n1","base_url":"http://127.0.0.1:43438","node_name":"127.0.0.1:43438_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//clusterstate.json/27)={
[junit4] > "pullReplicas":"0",
[junit4] > "replicationFactor":"1",
[junit4] > "shards":{"shard1":{
[junit4] > "range":"80000000-7fffffff",
[junit4] > "state":"active",
[junit4] > "replicas":{
[junit4] > "core_node4":{
[junit4] > "core":"c8n_1x3_lf_shard1_replica_n1",
[junit4] > "base_url":"http://127.0.0.1:43438",
[junit4] > "node_name":"127.0.0.1:43438_",
[junit4] > "state":"active",
[junit4] > "type":"NRT",
[junit4] > "leader":"true"},
[junit4] > "core_node5":{
[junit4] > "core":"c8n_1x3_lf_shard1_replica_n2",
[junit4] > "base_url":"http://127.0.0.1:45981",
[junit4] > "node_name":"127.0.0.1:45981_",
[junit4] > "state":"down",
[junit4] > "type":"NRT"},
[junit4] > "core_node6":{
[junit4] > "state":"down",
[junit4] > "base_url":"http://127.0.0.1:33388",
[junit4] > "core":"c8n_1x3_lf_shard1_replica_n3",
[junit4] > "node_name":"127.0.0.1:33388_",
[junit4] > "type":"NRT"}}}},
[junit4] > "router":{"name":"compositeId"},
[junit4] > "maxShardsPerNode":"1",
[junit4] > "autoAddReplicas":"false",
[junit4] > "nrtReplicas":"3",
[junit4] > "tlogReplicas":"0"}
[junit4] > at __randomizedtesting.SeedInfo.seed([49336704300C9DDB:C16758DE9EF0F023]:0)
[junit4] > at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
[junit4] > at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_49336704300C9DDB-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_t=PostingsFormat(name=LuceneVarGapFixedInterval), id=PostingsFormat(name=Direct), text=FSTOrd50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Memory), intDefault=DocValuesFormat(name=Memory), id_i1=DocValuesFormat(name=Asserting), range_facet_i_dv=DocValuesFormat(name=Memory), intDvoDefault=DocValuesFormat(name=Lucene70), range_facet_l=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1813, maxMBSortInHeap=5.1924247456840025, sim=RandomSimilarity(queryNorm=true): {}, locale=ar, timezone=America/Moncton
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=112237168,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [TestStressLiveNodes, TestClusterStateMutator, ChaosMonkeyNothingIsSafeWithPullReplicasTest, SearchHandlerTest, TestReplicationHandlerBackup, BlobRepositoryCloudTest, TestFieldCacheSortRandom, TestSort, HdfsDirectoryTest, TestSolrJ, TermVectorComponentTest, RecoveryAfterSoftCommitTest, CdcrBootstrapTest, DistributedMLTComponentTest, AnalysisAfterCoreReloadTest, TestCorePropertiesReload, TermVectorComponentDistributedTest, HdfsNNFailoverTest, TestFieldCollectionResource, CdcrReplicationHandlerTest, TestCrossCoreJoin, HttpPartitionTest, TestGroupingSearch, SynonymTokenizerTest, TestDelegationWithHadoopAuth, TestExportWriter, LargeFieldTest, DeleteStatusTest, TestCloudSchemaless, DistributedFacetPivotLargeTest, TestSolrConfigHandler, TestPerFieldSimilarity, TestLegacyNumericRangeQueryBuilder, TestPivotHelperCode, CollectionsAPIDistributedZkTest, DeleteReplicaTest, TestPostingsSolrHighlighter, TestIndexSearcher, ReplaceNodeTest, TestIntervalFaceting, TestHashPartitioner, TestUseDocValuesAsStored2, SSLMigrationTest, MultiThreadedOCPTest, TestQueryWrapperFilter, TestRandomFaceting, CustomHighlightComponentTest, TestConfigsApi, TestCloudPivotFacet, UpdateLogTest, ConnectionManagerTest, CacheHeaderTest, TestSerializedLuceneMatchVersion, TestCustomSort, TestFaceting, TestSystemIdResolver, AliasIntegrationTest, TestSolrIndexConfig, TestSolrCloudWithKerberosAlt, TestSQLHandlerNonCloud, TestApiFramework, DocValuesTest, TestHalfAndHalfDocValues, SimplePostToolTest, BasicAuthStandaloneTest, TestExceedMaxTermLength, TestMinMaxOnMultiValuedField, TestLockTree, TestCollectionsAPIViaSolrCloudCluster, TestSchemalessBufferedUpdates, TestPerFieldSimilarityWithDefaultOverride, QueryResultKeyTest, TestLRUStatsCache, TestExecutePlanAction, ConcurrentDeleteAndCreateCollectionTest, SpellPossibilityIteratorTest, TestNumericRangeQuery32, TestRecoveryHdfs, TestNumericTokenStream, TestFieldTypeCollectionResource, TestLargeCluster, TestSolrQueryResponse, TestUninvertingReader, LeaderInitiatedRecoveryOnCommitTest, OverseerCollectionConfigSetProcessorTest, NumberUtilsTest, TestWordDelimiterFilterFactory, FacetPivotSmallTest, TestReload, SolrCloudExampleTest, AnalysisErrorHandlingTest, TestFieldCacheWithThreads, RuleEngineTest, HdfsSyncSliceTest, TestJsonRequest, TestExactSharedStatsCache, TestMergePolicyConfig, TestZkChroot, TestSimpleTrackingShardHandler, TestFoldingMultitermQuery, QueryEqualityTest, SimpleFacetsTest, SpatialHeatmapFacetsTest, TestElisionMultitermQuery, SolrCoreCheckLockOnStartupTest, TestBM25SimilarityFactory, PropertiesRequestHandlerTest, TestConfig, TestComplexPhraseLeadingWildcard, PreAnalyzedUpdateProcessorTest, ShardSplitTest, TermsComponentTest, TestLRUCache, TestCSVLoader, TestHighFrequencyDictionaryFactory, ZkFailoverTest, TestSchemaSimilarityResource, BlockJoinFacetSimpleTest, SolrMetricReporterTest, TestPointFields, SolrShardReporterTest, CdcrVersionReplicationTest, PreAnalyzedFieldTest, TestDistribDocBasedVersion, ConjunctionSolrSpellCheckerTest, ZkNodePropsTest, TestGraphMLResponseWriter, HdfsAutoAddReplicasIntegrationTest, ExternalFileFieldSortTest, BasicFunctionalityTest, SOLR749Test, BasicZkTest, RankQueryTest, TestRemoteStreaming, RequestLoggingTest, TestBinaryResponseWriter, SegmentsInfoRequestHandlerTest, TestFastLRUCache, CollectionsAPIAsyncDistributedZkTest, ManagedSchemaRoundRobinCloudTest, FullHLLTest, TestQueryTypes, SolrJmxReporterCloudTest, AnalyticsQueryTest, SolrIndexConfigTest, CoreAdminHandlerTest, BlockDirectoryTest, OverseerRolesTest, LeaderFailoverAfterPartitionTest]
[junit4] Completed [351/765 (2!)] on J1 in 139.23s, 1 test, 1 failure <<< FAILURES!
[...truncated 49524 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 373 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/373/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
3 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction.testIntegration
Error Message:
Stack Trace:
java.util.ConcurrentModificationException
at __randomizedtesting.SeedInfo.seed([C48B7B3B457DBD72:74EA751760421C57]:0)
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.tearDown(SimSolrCloudTestCase.java:141)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeAddedTriggerRestoreState
Error Message:
Stack Trace:
java.util.ConcurrentModificationException
at __randomizedtesting.SeedInfo.seed([C48B7B3B457DBD72:4CB6F2447FBD5CDF]:0)
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.tearDown(SimSolrCloudTestCase.java:141)
at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testGammaDistribution
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([1918B1C95D2400E1:24629A677E5CAAF6]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testGammaDistribution(StreamExpressionTest.java:7808)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 11936 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestExecutePlanAction_C48B7B3B457DBD72-001/init-core-data-001
[junit4] 2> 678206 INFO (SUITE-TestExecutePlanAction-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 678206 INFO (SUITE-TestExecutePlanAction-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 678206 INFO (SUITE-TestExecutePlanAction-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 678230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 678230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 678230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10000_solr]
[junit4] 2> 678230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 678230 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 678235 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 678235 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
[junit4] 2> 678258 DEBUG (ScheduledTrigger-1711-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678260 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testExecute
[junit4] 2> 678260 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 678260 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 678263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 678263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 678263 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 678264 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 678264 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 678269 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 678269 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 678269 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 678270 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 678270 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 678270 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 678270 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 678270 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 678270 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678290 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678314 DEBUG (ScheduledTrigger-1714-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678337 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678358 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678373 DEBUG (simCloudManagerPool-1713-thread-1) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: testExecute
[junit4] 2> 678378 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678386 DEBUG (simCloudManagerPool-1713-thread-1) [ ] o.a.s.c.Assign Identify nodes using default
[junit4] 2> 678398 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678416 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction Collection ready after 1156ms
[junit4] 2> 678418 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678420 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.ExecutePlanAction -- processing event: {
[junit4] 2> "id":"1506a604cef519c0T5y74lvmt9hwilvifly8sh3hko",
[junit4] 2> "source":"mock_trigger_name",
[junit4] 2> "eventTime":1515080864239000000,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1515080864239000000],
[junit4] 2> "nodeNames":["127.0.0.1:10000_solr"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@57fba7cb, org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction$1@74d02a4d]}
[junit4] 2> 678420 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testExecute&targetNode=127.0.0.1:10001_solr&inPlaceMove=true&replica=core_node1
[junit4] 2> 678438 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678464 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678484 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678504 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678521 DEBUG (simCloudManagerPool-1713-thread-4) [ ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node4":{
[junit4] 2> "core":"testExecute_shard1_replica_n3",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testExecute",
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "type":"NRT"}}
[junit4] 2> 678524 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678550 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678574 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678596 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678616 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678636 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678639 DEBUG (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=OVERSEERSTATUS
[junit4] 2> 678643 DEBUG (simCloudManagerPool-1713-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testExecute / shard1: {"core_node2":{
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "core":"testExecute_shard1_replica_n2",
[junit4] 2> "leader":"true",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testExecute"}}
[junit4] 2> 678644 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction Collection ready after 7ms
[junit4] 2> 678644 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction -------------_ FINAL STATE --------------
[junit4] 2> 678644 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Node values: {
[junit4] 2> "127.0.0.1:10001_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":2,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":980,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10001_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10001,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
[junit4] 2> "127.0.0.1:10000_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":0,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":1000,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10000_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10000,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0}}
[junit4] 2> 678644 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Live nodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 678645 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Collection testExecute state: DocCollection(testExecute//clusterstate.json/3)={
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"1",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"2",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "autoCreated":"true",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "replicas":{
[junit4] 2> "core_node2":{
[junit4] 2> "core":"testExecute_shard1_replica_n2",
[junit4] 2> "leader":"true",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node4":{
[junit4] 2> "node_name":"127.0.0.1:10001_solr",
[junit4] 2> "core":"testExecute_shard1_replica_n3",
[junit4] 2> "state":"active",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "type":"NRT"}},
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active"}}}
[junit4] 2> 678645 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testExecute
[junit4] 2> 678645 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 678645 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 678645 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 678645 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 678645 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 678646 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 1
[junit4] 2> 678646 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 678647 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: [testExecute]
[junit4] 2> 678647 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 2
[junit4] 2> 678647 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 678647 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 2
[junit4] 2> 678647 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * testExecute 2
[junit4] 2> 678648 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - active 2
[junit4] 2> 678648 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 678648 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - CREATE 1
[junit4] 2> 678648 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - DELETESTATUS 2
[junit4] 2> 678648 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - MOVEREPLICA 1
[junit4] 2> 678648 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - OVERSEERSTATUS 1
[junit4] 2> 678648 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - REQUESTSTATUS 2
[junit4] 2> 678649 INFO (TEST-TestExecutePlanAction.testExecute-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 678658 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678680 DEBUG (ScheduledTrigger-1714-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678688 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testIntegration
[junit4] 2> 678688 DEBUG (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 678688 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 678688 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 678688 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 678688 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 678693 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 678693 DEBUG (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 678694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 678694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 678694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 678694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
[junit4] 2> 678694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 678694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 678694 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 678694 DEBUG (ScheduledTrigger-1717-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678715 DEBUG (ScheduledTrigger-1717-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678735 DEBUG (ScheduledTrigger-1717-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678760 DEBUG (ScheduledTrigger-1717-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678780 DEBUG (ScheduledTrigger-1717-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678800 DEBUG (ScheduledTrigger-1717-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678813 DEBUG (simCloudManagerPool-1716-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 678813 DEBUG (simCloudManagerPool-1716-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 678816 DEBUG (simCloudManagerPool-1716-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 678816 DEBUG (simCloudManagerPool-1716-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr, 127.0.0.1:10000_solr]
[junit4] 2> 678816 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 678817 DEBUG (simCloudManagerPool-1716-thread-2) [ ] o.a.s.c.o.ClusterStateMutator building a new cName: testIntegration
[junit4] 2> 678817 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 678817 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 678817 DEBUG (ScheduledTrigger-1717-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 678820 DEBUG (ScheduledTrigger-1717-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678827 DEBUG (simCloudManagerPool-1716-thread-2) [ ] o.a.s.c.Assign Identify nodes using default
[junit4] 2> 678841 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 678841 DEBUG (ScheduledTrigger-1717-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 678858 DEBUG (simCloudManagerPool-1716-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for testIntegration / shard1: {"core_node1":{
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "core":"testIntegration_shard1_replica_n1",
[junit4] 2> "leader":"true",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testIntegration"}}
[junit4] 2> 678862 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 678862 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
[junit4] 2> 678862 DEBUG (ScheduledTrigger-1717-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 678862 DEBUG (ScheduledTrigger-1717-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10001_solr
[junit4] 2> 678883 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 678883 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 678883 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10001_solr]
[junit4] 2> 678883 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a091c092378ecT5y74lvmt9hwilvifly8sh3hkp",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2824765784160492,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2824765784160492],
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}}
[junit4] 2> 678885 DEBUG (AutoscalingActionExecutor-1718-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a091c092378ecT5y74lvmt9hwilvifly8sh3hkp",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2824765784160492,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2824765784160492],
[junit4] 2> "_enqueue_time_":2824766842661792,
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}}
[junit4] 2> 678886 DEBUG (AutoscalingActionExecutor-1718-thread-1) [ ] o.a.s.c.a.ComputePlanAction -- processing event: {
[junit4] 2> "id":"a091c092378ecT5y74lvmt9hwilvifly8sh3hkp",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2824765784160492,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2824765784160492],
[junit4] 2> "_enqueue_time_":2824766842661792,
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}} with context properties: {BEFORE_ACTION=[compute_plan]}
[junit4] 2> 678887 INFO (AutoscalingActionExecutor-1718-thread-1) [ ] o.a.s.c.a.ComputePlanAction Computed Plan: action=MOVEREPLICA&collection=testIntegration&targetNode=127.0.0.1:10000_solr&inPlaceMove=true&replica=core_node2
[junit4] 2> 678887 INFO (AutoscalingActionExecutor-1718-thread-1) [ ] o.a.s.c.s.c.a.PolicyHelper returnSession, curr-time 2824767046 sessionWrapper.createTime 2824767035468342, this.sessionWrapper.createTime 2824767035468342
[junit4] 2> 678887 DEBUG (AutoscalingActionExecutor-1718-thread-1) [ ] o.a.s.c.a.ExecutePlanAction -- processing event: {
[junit4] 2> "id":"a091c092378ecT5y74lvmt9hwilvifly8sh3hkp",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":2824765784160492,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2824765784160492],
[junit4] 2> "_enqueue_time_":2824766842661792,
[junit4] 2> "nodeNames":["127.0.0.1:10001_solr"]}} with context properties: {operations=[org.apache.solr.client.solrj.request.CollectionAdminRequest$MoveReplica@760b1e58], AFTER_ACTION=[compute_plan], BEFORE_ACTION=[compute_plan, execute_plan]}
[junit4] 2> 678887 DEBUG (AutoscalingActionExecutor-1718-thread-1) [ ] o.a.s.c.a.ExecutePlanAction Executing operation: action=MOVEREPLICA&collection=testIntegration&targetNode=127.0.0.1:10000_solr&inPlaceMove=true&replica=core_node2
[junit4] 2> 678994 DEBUG (simCloudManagerPool-1716-thread-7) [ ] o.a.s.c.a.s.SimClusterStateProvider -- new replica: {"core_node4":{
[junit4] 2> "core":"testIntegration_shard1_replica_n3",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testIntegration",
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "type":"NRT"}}
[junit4] 2> 679111 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction -------------_ FINAL STATE --------------
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Node values: {
[junit4] 2> "127.0.0.1:10001_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":1,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":970,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10001_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10001,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0},
[junit4] 2> "127.0.0.1:10000_solr":{
[junit4] 2> "sysprop.java.vendor":"Oracle Corporation",
[junit4] 2> "cores":2,
[junit4] 2> "host":"127.0.0.1",
[junit4] 2> "freedisk":980,
[junit4] 2> "sysprop.java.version":"1.8.0_152",
[junit4] 2> "ip_1":"127",
[junit4] 2> "ip_3":"0",
[junit4] 2> "ip_2":"0",
[junit4] 2> "ip_4":"1",
[junit4] 2> "node":"127.0.0.1:10000_solr",
[junit4] 2> "metrics:solr.jvm:buffers.direct.Count":0,
[junit4] 2> "port":10000,
[junit4] 2> "heapUsage":123450000,
[junit4] 2> "sysLoadAvg":1.0,
[junit4] 2> "metrics:solr.node:ADMIN./admin/authorization.clientErrors:count":0}}
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Live nodes: [127.0.0.1:10000_solr]
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestExecutePlanAction * Collection testIntegration state: DocCollection(testIntegration//clusterstate.json/10)={
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "pullReplicas":"0",
[junit4] 2> "router":{"name":"compositeId"},
[junit4] 2> "maxShardsPerNode":"1",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"2",
[junit4] 2> "tlogReplicas":"0",
[junit4] 2> "autoCreated":"true",
[junit4] 2> "shards":{"shard1":{
[junit4] 2> "replicas":{
[junit4] 2> "core_node1":{
[junit4] 2> "core":"testIntegration_shard1_replica_n1",
[junit4] 2> "leader":"true",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT"},
[junit4] 2> "core_node4":{
[junit4] 2> "node_name":"127.0.0.1:10000_solr",
[junit4] 2> "core":"testIntegration_shard1_replica_n3",
[junit4] 2> "state":"active",
[junit4] 2> "INDEX.sizeInBytes":123450000,
[junit4] 2> "type":"NRT"}},
[junit4] 2> "range":"80000000-7fffffff",
[junit4] 2> "state":"active"}}}
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testIntegration
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 1
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 0
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10001_solr
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: [testIntegration]
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 2
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 2
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 2
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * testIntegration 2
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - active 2
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 679112 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - CREATE 1
[junit4] 2> 679113 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - DELETESTATUS 1
[junit4] 2> 679113 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - MOVEREPLICA 1
[junit4] 2> 679113 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - REQUESTSTATUS 1
[junit4] 2> 679113 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 1
[junit4] 2> 679113 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 1
[junit4] 2> 679113 INFO (TEST-TestExecutePlanAction.testIntegration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 679113 DEBUG (AutoscalingActionExecutor-1718-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 11382 ms for event id=a091c092378ecT5y74lvmt9hwilvifly8sh3hkp
[junit4] 2> 679114 DEBUG (ScheduledTrigger-1717-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 679116 DEBUG (ScheduledTrigger-1717-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestExecutePlanAction -Dtests.method=testIntegration -Dtests.seed=C48B7B3B457DBD72 -Dtests.slow=true -Dtests.locale=ar-QA -Dtests.timezone=Africa/Gaborone -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.47s J1 | TestExecutePlanAction.testIntegration <<<
[junit4] > Throwable #1: java.util.ConcurrentModificationException
[junit4] > at __randomizedtesting.SeedInfo.seed([C48B7B3B457DBD72:74EA751760421C57]:0)
[junit4] > at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
[junit4] > at java.util.ArrayList$Itr.next(ArrayList.java:859)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.tearDown(SimSolrCloudTestCase.java:141)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 679116 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 679120 DEBUG (SUITE-TestExecutePlanAction-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestExecutePlanAction_C48B7B3B457DBD72-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1709, maxMBSortInHeap=5.080522274153202, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-QA, timezone=Africa/Gaborone
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=144327640,total=323878912
[junit4] 2> NOTE: All tests run in this JVM: [TestLRUCache, AddSchemaFieldsUpdateProcessorFactoryTest, TestMacroExpander, CloneFieldUpdateProcessorFactoryTest, AnalyticsQueryTest, SolrCoreTest, BlockJoinFacetDistribTest, JavabinLoaderTest, JvmMetricsTest, TestCharFilters, TestSolrCoreParser, TriggerIntegrationTest, PropertiesRequestHandlerTest, SpellingQueryConverterTest, TestReload, ChaosMonkeySafeLeaderTest, LargeFieldTest, ReplaceNodeTest, DocumentAnalysisRequestHandlerTest, TestAnalyzeInfixSuggestions, AssignTest, TestCorePropertiesReload, DebugComponentTest, CdcrVersionReplicationTest, SolrSlf4jReporterTest, TestPerFieldSimilarityWithDefaultOverride, TestConfigSetsAPIExclusivity, TestFieldCacheWithThreads, SystemInfoHandlerTest, HdfsAutoAddReplicasIntegrationTest, TestSolrIndexConfig, TestIndexSearcher, QueryEqualityTest, TestLockTree, CollectionsAPIAsyncDistributedZkTest, DocValuesTest, TestFileDictionaryLookup, TestFastOutputStream, DistributedFacetPivotSmallAdvancedTest, HdfsBasicDistributedZkTest, AutoAddReplicasPlanActionTest, TestCryptoKeys, TestFieldCacheSortRandom, SpellPossibilityIteratorTest, TestExecutePlanAction]
[junit4] Completed [99/765 (1!)] on J1 in 0.95s, 2 tests, 1 error <<< FAILURES!
[...truncated 735 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_C48B7B3B457DBD72-001/init-core-data-001
[junit4] 2> 1839217 WARN (SUITE-TestTriggerIntegration-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 1839217 INFO (SUITE-TestTriggerIntegration-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1839218 INFO (SUITE-TestTriggerIntegration-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1839218 INFO (SUITE-TestTriggerIntegration-seed#[C48B7B3B457DBD72]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1839219 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1839219 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 1839219 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839219 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 1839219 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 1839221 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTriggerRestoreState
[junit4] 2> 1839221 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 1839221 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1839221 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839221 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1839221 DEBUG (ScheduledTrigger-5649-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839222 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1839223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1839223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 1839223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 1839223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1839223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839223 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1839223 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839243 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839266 DEBUG (ScheduledTrigger-5652-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839286 DEBUG (ScheduledTrigger-5652-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839306 DEBUG (ScheduledTrigger-5652-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839325 DEBUG (simCloudManagerPool-5651-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1839325 DEBUG (simCloudManagerPool-5651-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 1839325 DEBUG (simCloudManagerPool-5651-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839325 DEBUG (simCloudManagerPool-5651-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=5, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1839326 DEBUG (simCloudManagerPool-5651-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839326 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1839326 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1839326 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1839326 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1839326 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1839326 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10007_solr at time 2871230138595399
[junit4] 2> 1839326 DEBUG (ScheduledTrigger-5652-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1839347 DEBUG (ScheduledTrigger-5652-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1839347 DEBUG (ScheduledTrigger-5652-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1839347 DEBUG (ScheduledTrigger-5652-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1839357 DEBUG (simCloudManagerPool-5651-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1839357 DEBUG (simCloudManagerPool-5651-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 1839357 DEBUG (simCloudManagerPool-5651-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1839357 DEBUG (simCloudManagerPool-5651-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 1839357 DEBUG (simCloudManagerPool-5651-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 127.0.0.1:10007_solr]
[junit4] 2> 1839357 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1839357 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1839357 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:10007_solr
[junit4] 2> 1839357 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1839358 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_restore_trigger
[junit4] 2> 1839358 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1839358 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_restore_trigger firing registered processor for nodes: [127.0.0.1:10007_solr] added at times [2871231699896449], now=2871231712000599
[junit4] 2> 1839358 DEBUG (ScheduledTrigger-5652-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a335eb98a4881T5y74lvmt9hwilvifly8sh3oeu",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":2871231699896449,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2871231699896449],
[junit4] 2> "nodeNames":["127.0.0.1:10007_solr"]}}
[junit4] 2> 1839358 DEBUG (AutoscalingActionExecutor-5653-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a335eb98a4881T5y74lvmt9hwilvifly8sh3oeu",
[junit4] 2> "source":"node_added_restore_trigger",
[junit4] 2> "eventTime":2871231699896449,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2871231699896449],
[junit4] 2> "_enqueue_time_":2871231718427699,
[junit4] 2> "nodeNames":["127.0.0.1:10007_solr"]}}
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTriggerRestoreState
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 3
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 3
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 1
[junit4] 2> 1839359 INFO (TEST-TestTriggerIntegration.testNodeAddedTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1839359 DEBUG (AutoscalingActionExecutor-5653-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 43 ms for event id=a335eb98a4881T5y74lvmt9hwilvifly8sh3oeu
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestTriggerIntegration -Dtests.method=testNodeAddedTriggerRestoreState -Dtests.seed=C48B7B3B457DBD72 -Dtests.slow=true -Dtests.locale=id-ID -Dtests.timezone=America/Guayaquil -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.14s J0 | TestTriggerIntegration.testNodeAddedTriggerRestoreState <<<
[junit4] > Throwable #1: java.util.ConcurrentModificationException
[junit4] > at __randomizedtesting.SeedInfo.seed([C48B7B3B457DBD72:4CB6F2447FBD5CDF]:0)
[junit4] > at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
[junit4] > at java.util.ArrayList$Itr.next(ArrayList.java:859)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.SimSolrCloudTestCase.tearDown(SimSolrCloudTestCase.java:141)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1839367 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
[junit4] 2> 1839367 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 1839368 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1839369 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1839369 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1839370 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1839370 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 1839370 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839370 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
[junit4] 2> 1839370 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 1839370 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839370 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 1839370 DEBUG (ScheduledTrigger-5655-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839390 DEBUG (ScheduledTrigger-5655-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839411 DEBUG (ScheduledTrigger-5655-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839433 DEBUG (ScheduledTrigger-5655-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839453 DEBUG (ScheduledTrigger-5655-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839473 DEBUG (simCloudManagerPool-5654-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1839473 DEBUG (simCloudManagerPool-5654-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 1839473 DEBUG (ScheduledTrigger-5655-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 1839474 DEBUG (simCloudManagerPool-5654-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1839474 DEBUG (simCloudManagerPool-5654-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr, 127.0.0.1:10008_solr]
[junit4] 2> 1839474 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 1839474 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1839474 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839474 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 1839474 DEBUG (ScheduledTrigger-5655-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 1839474 DEBUG (ScheduledTrigger-5655-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10008_solr
[junit4] 2> 1839493 DEBUG (ScheduledTrigger-5655-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839494 DEBUG (ScheduledTrigger-5655-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10008_solr
[junit4] 2> 1839502 DEBUG (ScheduledTrigger-5655-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 1839504 DEBUG (simCloudManagerPool-5654-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1839505 DEBUG (simCloudManagerPool-5654-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 1839505 DEBUG (simCloudManagerPool-5654-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839505 DEBUG (simCloudManagerPool-5654-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839505 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 1839505 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 1839505 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839505 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 1839505 DEBUG (ScheduledTrigger-5655-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 1839505 DEBUG (ScheduledTrigger-5655-thread-3) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10008_solr]
[junit4] 2> 1839505 DEBUG (ScheduledTrigger-5655-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"a336015e5dbfdT5y74lvmt9hwilvifly8sh3oex",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":2871237544369149,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2871237544369149],
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"]}}
[junit4] 2> 1839506 DEBUG (AutoscalingActionExecutor-5656-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"a336015e5dbfdT5y74lvmt9hwilvifly8sh3oex",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":2871237544369149,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[2871237544369149],
[junit4] 2> "_enqueue_time_":2871239094614549,
[junit4] 2> "nodeNames":["127.0.0.1:10008_solr"]}}
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 1839506 DEBUG (AutoscalingActionExecutor-5656-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 22 ms for event id=a336015e5dbfdT5y74lvmt9hwilvifly8sh3oex
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 2
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 2
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 1839506 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10008_solr
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 2
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_restore_trigger
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 1839507 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 1839511 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
[junit4] 2> 1839511 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 1839511 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 1839511 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10008_solr
[junit4] 2> 1839511 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1839512 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1839512 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839512 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1839513 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1839513 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1839514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 1839514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10005_solr]
[junit4] 2> 1839514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
[junit4] 2> 1839514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 1839514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
[junit4] 2> 1839514 DEBUG (ScheduledTrigger-5658-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839538 DEBUG (ScheduledTrigger-5658-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839559 DEBUG (ScheduledTrigger-5658-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839579 DEBUG (ScheduledTrigger-5658-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839599 DEBUG (ScheduledTrigger-5658-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839619 DEBUG (ScheduledTrigger-5658-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839620 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 1
[junit4] 2> 1839620 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 1839620 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1839621 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 1839621 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 1839622 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1839622 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
[junit4] 2> 1839622 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 1839622 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10005_solr
[junit4] 2> 1839622 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1839622 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
[junit4] 2> 1839622 DEBUG (ScheduledTrigger-5661-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839622 DEBUG (ScheduledTrigger-5661-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10005_solr
[junit4] 2> 1839642 DEBUG (ScheduledTrigger-5661-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839662 DEBUG (ScheduledTrigger-5661-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839693 DEBUG (ScheduledTrigger-5661-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839722 DEBUG (ScheduledTrigger-5661-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1839722 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD TRIGGERS
[junit4] 2> 1839724 DEBUG (simCloudManagerPool-5660-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1839724 DEBUG (simCloudManagerPool-5660-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 1839725 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 1839725 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 1839725 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
[junit4] 2> 1839725 DEBUG (ScheduledTrigger-5661-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1839725 DEBUG (ScheduledTrigger-5661-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1839725 DEBUG (simCloudManagerPool-5660-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10006_solr, 127.0.0.1:10009_solr]
[junit4] 2> 1839726 DEBUG (simCloudManagerPool-5660-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 1839726 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[C48B7B3B457DBD72]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD NODE 1
[junit4] 2> 1839726 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
[junit4] 2> 1839726 INFO (Simu
[...truncated too long message...]
Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 143258 INFO (jetty-closer-11-thread-4) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@757dfd3: rootName = solr_60604, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4a7bb9f9
[junit4] 2> 143260 INFO (zkCallback-42-thread-2-processing-n:127.0.0.1:60604_solr) [n:127.0.0.1:60604_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60604_solr
[junit4] 2> 143261 INFO (jetty-closer-11-thread-4) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:60604_solr
[junit4] 2> 143263 INFO (zkCallback-42-thread-2-processing-n:127.0.0.1:60604_solr) [n:127.0.0.1:60604_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
[junit4] 2> 143263 INFO (jetty-closer-11-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7d7f82c{/solr,null,UNAVAILABLE}
[junit4] 2> 143265 INFO (zkCallback-38-thread-1-processing-n:127.0.0.1:34995_solr) [n:127.0.0.1:34995_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
[junit4] 2> 143267 INFO (jetty-closer-11-thread-4) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 143270 INFO (zkCallback-44-thread-2-processing-n:127.0.0.1:50146_solr) [n:127.0.0.1:50146_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
[junit4] 2> 143319 INFO (jetty-closer-11-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 143319 INFO (jetty-closer-11-thread-3) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@588bfbc3: rootName = solr_34995, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4a7bb9f9
[junit4] 2> 143320 INFO (zkCallback-38-thread-1-processing-n:127.0.0.1:34995_solr) [n:127.0.0.1:34995_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34995_solr
[junit4] 2> 143322 INFO (jetty-closer-11-thread-3) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:34995_solr
[junit4] 2> 143323 INFO (jetty-closer-11-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 143324 INFO (zkCallback-38-thread-2-processing-n:127.0.0.1:34995_solr) [n:127.0.0.1:34995_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 143325 INFO (zkCallback-44-thread-2-processing-n:127.0.0.1:50146_solr) [n:127.0.0.1:50146_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 143325 INFO (coreCloseExecutor-274-thread-1) [n:127.0.0.1:34995_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.c.SolrCore [collection1_shard2_replica_n3] CLOSING SolrCore org.apache.solr.core.SolrCore@1681d416
[junit4] 2> 143325 INFO (coreCloseExecutor-274-thread-1) [n:127.0.0.1:34995_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard2.replica_n3, tag=377607190
[junit4] 2> 143325 INFO (coreCloseExecutor-274-thread-1) [n:127.0.0.1:34995_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3d610cca: rootName = solr_34995, domain = solr.core.collection1.shard2.replica_n3, service url = null, agent id = null] for registry solr.core.collection1.shard2.replica_n3 / com.codahale.metrics.MetricRegistry@4e9665be
[junit4] 2> 143334 INFO (coreCloseExecutor-274-thread-1) [n:127.0.0.1:34995_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=377607190
[junit4] 2> 143335 INFO (jetty-closer-11-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@410a5ffa{/solr,null,UNAVAILABLE}
[junit4] 2> 143339 INFO (jetty-closer-11-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 143339 INFO (jetty-closer-11-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@305ef: rootName = solr_50146, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@4e4434e9
[junit4] 2> 143343 INFO (jetty-closer-11-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 143343 INFO (jetty-closer-11-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@69f2d18b: rootName = solr_50146, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4a7bb9f9
[junit4] 2> 143345 INFO (jetty-closer-11-thread-1) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50146_solr
[junit4] 2> 143346 INFO (jetty-closer-11-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 143346 INFO (zkCallback-44-thread-2-processing-n:127.0.0.1:50146_solr) [n:127.0.0.1:50146_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 144860 WARN (zkCallback-38-thread-2-processing-n:127.0.0.1:34995_solr) [n:127.0.0.1:34995_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 144861 INFO (jetty-closer-11-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@614b4ee6{/solr,null,UNAVAILABLE}
[junit4] 2> 144862 INFO (jetty-closer-11-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@e84c30{/solr,null,UNAVAILABLE}
[junit4] 2> 144865 ERROR (SUITE-StreamExpressionTest-seed#[1918B1C95D2400E1]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 144867 INFO (SUITE-StreamExpressionTest-seed#[1918B1C95D2400E1]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53399 53399
[junit4] 2> 145021 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
[junit4] 2> java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:182)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 145021 INFO (Thread-52) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53399 53399
[junit4] 2> 145064 WARN (Thread-52) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 99 /solr/aliases.json
[junit4] 2> 17 /solr/configs/conf
[junit4] 2> 8 /solr/configs/ml
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 99 /solr/clusterprops.json
[junit4] 2> 99 /solr/clusterstate.json
[junit4] 2> 38 /solr/collections/destinationCollection/state.json
[junit4] 2> 26 /solr/collections/parallelDestinationCollection1/state.json
[junit4] 2> 20 /solr/collections/parallelDestinationCollection/state.json
[junit4] 2> 10 /solr/collections/mainCorpus1/state.json
[junit4] 2> 10 /solr/collections/destination/state.json
[junit4] 2> 10 /solr/collections/destination1/state.json
[junit4] 2> 10 /solr/collections/collection1/state.json
[junit4] 2> 10 /solr/collections/workQueue1/state.json
[junit4] 2> 10 /solr/collections/workQueue/state.json
[junit4] 2> 10 /solr/collections/uknownCollection/state.json
[junit4] 2> 10 /solr/collections/modelCollection/state.json
[junit4] 2> 10 /solr/collections/checkpointCollection/state.json
[junit4] 2> 10 /solr/collections/mainCorpus/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 99 /solr/live_nodes
[junit4] 2> 99 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_1918B1C95D2400E1-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=fr-CH, timezone=Etc/GMT+12
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_152 (64-bit)/cpus=3,threads=1,free=97033240,total=227909632
[junit4] 2> NOTE: All tests run in this JVM: [FacetFieldTest, HttpSolrClientConPoolTest, CorrelationEvaluatorTest, CubedRootEvaluatorTest, ArcCosineEvaluatorTest, SolrSchemalessExampleTest, ModuloEvaluatorTest, StreamExpressionParserTest, TestPathTrie, AndEvaluatorTest, LargeVolumeJettyTest, StreamExpressionToExplanationTest, SolrParamTest, StreamExpressionTest]
[junit4] Completed [70/146 (1!)] on J1 in 129.45s, 148 tests, 1 failure <<< FAILURES!
[...truncated 47457 lines...]