You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2016/07/03 02:36:36 UTC

[JENKINS] Lucene-Solr-Tests-6.x - Build # 308 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/308/

5 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Expected to find shardAddress in the up shard info

Stack Trace:
java.lang.AssertionError: Expected to find shardAddress in the up shard info
	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A:1F04C9C5A280E482]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1162)
	at org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1103)
	at org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:963)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1018)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A]:0)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelay

Error Message:
expected:<2> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<1>
	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A:E8CE419A651EA4F0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:130)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelay(ZkStateReaderTest.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.overseer.ZkStateWriterTest.testExternalModificationToStateFormat2

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A:E68DE2CB6DD775FA]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.overseer.ZkStateWriterTest.testExternalModificationToStateFormat2(ZkStateWriterTest.java:322)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10659 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateWriterTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.overseer.ZkStateWriterTest_9750F61F0C7C897A-001/init-core-data-001
   [junit4]   2> 76009 INFO  (SUITE-ZkStateWriterTest-seed#[9750F61F0C7C897A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 76011 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZkStateWriterBatching
   [junit4]   2> 76011 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 76012 INFO  (Thread-379) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 76012 INFO  (Thread-379) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 76112 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:60836
   [junit4]   2> 76113 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76117 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76119 INFO  (zkCallback-116-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@f0c4089 name:ZooKeeperConnection Watcher:127.0.0.1:60836 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76119 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76120 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76121 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76125 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76126 INFO  (zkCallback-117-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c6eade name:ZooKeeperConnection Watcher:127.0.0.1:60836 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76126 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76126 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76126 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 76130 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76152 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76161 INFO  (zkCallback-118-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@41288d50 name:ZooKeeperConnection Watcher:127.0.0.1:60836/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76161 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76161 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76162 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 76163 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 76163 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 76164 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 76165 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 76165 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 76166 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 76166 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 76167 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 76168 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 76169 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 76170 INFO  (zkCallback-118-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 76170 WARN  (zkCallback-118-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 76171 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60836 60836
   [junit4]   2> 76335 INFO  (Thread-379) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60836 60836
   [junit4]   2> 76336 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZkStateWriterBatching
   [junit4]   2> 76339 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSingleLegacyCollection
   [junit4]   2> 76339 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 76339 INFO  (Thread-380) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 76339 INFO  (Thread-380) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 76439 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:47481
   [junit4]   2> 76439 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76440 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76444 INFO  (zkCallback-120-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@528e32d7 name:ZooKeeperConnection Watcher:127.0.0.1:47481 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76444 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76444 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76445 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76448 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76449 INFO  (zkCallback-121-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@66bbdc8b name:ZooKeeperConnection Watcher:127.0.0.1:47481 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76449 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76449 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76449 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 76450 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76451 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76452 INFO  (zkCallback-122-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@209d2a3a name:ZooKeeperConnection Watcher:127.0.0.1:47481/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76452 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76452 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76453 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 76454 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 76455 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 76455 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 76456 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 76457 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 76457 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 76458 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 76459 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 76460 INFO  (zkCallback-122-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 76464 INFO  (zkCallback-122-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 76466 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47481 47481
   [junit4]   2> 76607 INFO  (Thread-380) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47481 47481
   [junit4]   2> 76608 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSingleLegacyCollection
   [junit4]   2> 76612 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToStateFormat2
   [junit4]   2> 76612 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 76612 INFO  (Thread-381) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 76612 INFO  (Thread-381) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 76712 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:40414
   [junit4]   2> 76713 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76721 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76740 INFO  (zkCallback-124-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@61e58d6 name:ZooKeeperConnection Watcher:127.0.0.1:40414 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76744 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76744 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76780 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76792 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76804 INFO  (zkCallback-125-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@12405cc9 name:ZooKeeperConnection Watcher:127.0.0.1:40414 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76805 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76806 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76806 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 76823 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155ae17f5c60001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 76823 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 76840 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 76847 INFO  (zkCallback-126-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@295502e3 name:ZooKeeperConnection Watcher:127.0.0.1:40414/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 76848 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 76848 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 76864 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 76884 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 76886 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 76887 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 76887 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 76888 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 76888 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 76889 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 76889 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 76890 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 76891 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c2/state.json
   [junit4]   2> 76897 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155ae17f5c60002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 76901 INFO  (zkCallback-126-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 76901 WARN  (zkCallback-126-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 76920 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40414 40414
   [junit4]   2> 77088 INFO  (Thread-381) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40414 40414
   [junit4]   2> 77092 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToStateFormat2
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateWriterTest -Dtests.method=testExternalModificationToStateFormat2 -Dtests.seed=9750F61F0C7C897A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=el-CY -Dtests.timezone=EST5EDT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.49s J1 | ZkStateWriterTest.testExternalModificationToStateFormat2 <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A:E68DE2CB6DD775FA]:0)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateWriterTest.testExternalModificationToStateFormat2(ZkStateWriterTest.java:322)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 77102 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToSharedClusterState
   [junit4]   2> 77102 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 77112 INFO  (Thread-382) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 77112 INFO  (Thread-382) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 77210 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:35985
   [junit4]   2> 77210 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 77221 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 77236 INFO  (zkCallback-128-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@775b5631 name:ZooKeeperConnection Watcher:127.0.0.1:35985 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77236 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 77236 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 77237 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155ae17f7bd0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 77237 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 77238 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 77238 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5cb3f187 name:ZooKeeperConnection Watcher:127.0.0.1:35985 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77238 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 77238 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 77238 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 77241 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 77241 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 77242 INFO  (zkCallback-130-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2b845ec9 name:ZooKeeperConnection Watcher:127.0.0.1:35985/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77242 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 77242 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 77243 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 77244 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 77244 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 77245 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 77246 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 77247 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 77247 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 77248 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 77248 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 77249 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 77250 INFO  (zkCallback-130-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 77252 INFO  (zkCallback-130-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 77256 INFO  (zkCallback-130-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 77256 WARN  (zkCallback-130-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 77256 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35985 35985
   [junit4]   2> 77497 INFO  (Thread-382) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35985 35985
   [junit4]   2> 77498 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToSharedClusterState
   [junit4]   2> 77501 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSingleExternalCollection
   [junit4]   2> 77502 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 77502 INFO  (Thread-383) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 77502 INFO  (Thread-383) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 77602 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:44510
   [junit4]   2> 77602 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 77603 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 77613 INFO  (zkCallback-132-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@11ffde01 name:ZooKeeperConnection Watcher:127.0.0.1:44510 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77613 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 77613 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 77614 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 77615 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 77616 INFO  (zkCallback-133-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@790da343 name:ZooKeeperConnection Watcher:127.0.0.1:44510 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77616 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 77616 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 77616 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 77618 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 77618 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 77619 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5cfb8e27 name:ZooKeeperConnection Watcher:127.0.0.1:44510/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77619 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 77620 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 77620 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 77621 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 77622 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 77622 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 77623 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 77624 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 77624 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 77624 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 77625 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 77626 INFO  (zkCallback-134-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 77626 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 77630 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44510 44510
   [junit4]   2> 77793 INFO  (Thread-383) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44510 44510
   [junit4]   2> 77793 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSingleExternalCollection
   [junit4]   2> 77794 INFO  (SUITE-ZkStateWriterTest-seed#[9750F61F0C7C897A]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.overseer.ZkStateWriterTest_9750F61F0C7C897A-001
   [junit4]   2> Jul 03, 2016 12:10:54 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=ClassicSimilarity, locale=el-CY, timezone=EST5EDT
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=231598808,total=444071936
   [junit4]   2> NOTE: All tests run in this JVM: [TestCloudInspectUtil, HdfsNNFailoverTest, OpenCloseCoreStressTest, CoreAdminCreateDiscoverTest, HdfsTlogReplayBufferedWhileIndexingTest, TestCoreContainer, ZkControllerTest, SSLMigrationTest, FileUtilsTest, DistributedTermsComponentTest, TestExceedMaxTermLength, ZkSolrClientTest, HdfsChaosMonkeyNothingIsSafeTest, ZkStateWriterTest]
   [junit4] Completed [15/619 (1!)] on J1 in 2.05s, 5 tests, 1 failure <<< FAILURES!

[...truncated 221 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateReaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_9750F61F0C7C897A-001/init-core-data-001
   [junit4]   2> 294561 INFO  (SUITE-ZkStateReaderTest-seed#[9750F61F0C7C897A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 294562 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> 294562 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 294574 INFO  (Thread-1210) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 294574 INFO  (Thread-1210) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 294674 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:37705
   [junit4]   2> 294674 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 294693 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 294708 INFO  (zkCallback-382-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@212f101 name:ZooKeeperConnection Watcher:127.0.0.1:37705 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 294708 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 294708 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 294709 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 294731 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 294753 INFO  (zkCallback-383-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@268c2015 name:ZooKeeperConnection Watcher:127.0.0.1:37705 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 294753 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 294753 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 294753 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 294756 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 294781 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 294793 INFO  (zkCallback-384-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4022ab3e name:ZooKeeperConnection Watcher:127.0.0.1:37705/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 294793 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 294793 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 294793 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 294794 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 294796 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 294796 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 294797 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 294797 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 294798 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 294798 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 294800 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 294800 INFO  (zkCallback-384-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 294802 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 294818 INFO  (zkCallback-384-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 294822 WARN  (zkCallback-384-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /clusterstate.json]
   [junit4]   2> 294822 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37705 37705
   [junit4]   2> 295068 INFO  (Thread-1210) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37705 37705
   [junit4]   2> 295069 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> 295071 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelay
   [junit4]   2> 295071 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 295076 INFO  (Thread-1211) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 295076 INFO  (Thread-1211) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 295176 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:38286
   [junit4]   2> 295177 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295177 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295195 INFO  (zkCallback-386-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28b0e9b2 name:ZooKeeperConnection Watcher:127.0.0.1:38286 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295195 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295195 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295197 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295215 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295227 INFO  (zkCallback-387-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@61b11724 name:ZooKeeperConnection Watcher:127.0.0.1:38286 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295227 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295228 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295228 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 295229 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295230 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295237 INFO  (zkCallback-388-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5572b418 name:ZooKeeperConnection Watcher:127.0.0.1:38286/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295237 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295237 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295237 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 295238 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 295238 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 295239 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 295239 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 295240 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 295240 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 295240 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 295241 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 295241 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 295245 INFO  (zkCallback-388-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 295252 INFO  (zkCallback-388-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 295253 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 295254 INFO  (zkCallback-388-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 295254 INFO  (zkCallback-388-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/c1/state.json] for collection [c1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 295255 INFO  (zkCallback-388-thread-2) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 295257 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38286 38286
   [junit4]   2> 295413 INFO  (Thread-1211) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38286 38286
   [junit4]   2> 295414 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelay
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateReaderTest -Dtests.method=testStateFormatUpdateWithTimeDelay -Dtests.seed=9750F61F0C7C897A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=it-CH -Dtests.timezone=America/Indiana/Indianapolis -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.35s J0 | ZkStateReaderTest.testStateFormatUpdateWithTimeDelay <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<2> but was:<1>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A:E8CE419A651EA4F0]:0)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:130)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelay(ZkStateReaderTest.java:53)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 295417 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalCollectionWatchedNotWatched
   [junit4]   2> 295417 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 295421 INFO  (Thread-1212) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 295421 INFO  (Thread-1212) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 295522 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:48176
   [junit4]   2> 295522 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295523 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295532 INFO  (zkCallback-390-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@792c4bc name:ZooKeeperConnection Watcher:127.0.0.1:48176 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295532 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295532 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295533 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295534 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295539 INFO  (zkCallback-391-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2e65fc6c name:ZooKeeperConnection Watcher:127.0.0.1:48176 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295539 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295539 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295540 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 295542 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295542 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295543 INFO  (zkCallback-392-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c034e29 name:ZooKeeperConnection Watcher:127.0.0.1:48176/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295543 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295544 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295544 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 295545 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 295546 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 295546 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 295547 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 295547 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 295548 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 295548 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 295549 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 295549 INFO  (zkCallback-392-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 295549 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 295550 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 295559 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48176 48176
   [junit4]   2> 295711 INFO  (Thread-1212) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48176 48176
   [junit4]   2> 295712 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalCollectionWatchedNotWatched
   [junit4]   2> 295714 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> 295714 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 295714 INFO  (Thread-1213) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 295714 INFO  (Thread-1213) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 295814 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:44085
   [junit4]   2> 295814 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295820 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295823 INFO  (zkCallback-394-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1ed4c27f name:ZooKeeperConnection Watcher:127.0.0.1:44085 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295823 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295823 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295824 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295825 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295837 INFO  (zkCallback-395-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@22aa4cd0 name:ZooKeeperConnection Watcher:127.0.0.1:44085 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295837 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295837 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295837 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 295839 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 295841 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 295845 INFO  (zkCallback-396-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@54eb0358 name:ZooKeeperConnection Watcher:127.0.0.1:44085/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 295845 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 295845 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 295845 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 295846 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 295846 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 295847 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 295848 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 295848 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 295848 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 295849 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 295850 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 295853 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 295853 INFO  (zkCallback-396-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 295854 INFO  (zkCallback-396-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 295855 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 295855 INFO  (zkCallback-396-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/c1/state.json] for collection [c1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 295855 INFO  (zkCallback-396-thread-1) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 295855 INFO  (zkCallback-396-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 295860 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44085 44085
   [junit4]   2> 295959 INFO  (Thread-1213) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44085 44085
   [junit4]   2> 295964 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> 295966 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWatchedCollectionCreation
   [junit4]   2> 295966 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 295977 INFO  (Thread-1214) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 295977 INFO  (Thread-1214) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 296082 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:53286
   [junit4]   2> 296082 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 296090 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 296110 INFO  (zkCallback-398-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@78268a20 name:ZooKeeperConnection Watcher:127.0.0.1:53286 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296110 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 296110 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 296113 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 296116 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 296125 INFO  (zkCallback-399-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c663dd5 name:ZooKeeperConnection Watcher:127.0.0.1:53286 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296125 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 296125 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 296125 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 296126 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 296153 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 296177 INFO  (zkCallback-400-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3403252a name:ZooKeeperConnection Watcher:127.0.0.1:53286/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296177 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 296177 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 296177 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 296178 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 296178 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 296179 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 296179 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 296180 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 296180 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 296180 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 296181 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 296189 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 296189 INFO  (zkCallback-400-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 296190 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 296190 INFO  (zkCallback-400-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/c1/state.json] for collection [c1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 296191 INFO  (zkCallback-400-thread-1) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 296197 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53286 53286
   [junit4]   2> 296361 INFO  (Thread-1214) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53286 53286
   [junit4]   2> 296363 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWatchedCollectionCreation
   [junit4]   2> 296378 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 296378 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 296381 INFO  (Thread-1215) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 296381 INFO  (Thread-1215) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 296481 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.ZkTestServer start zk server on port:34436
   [junit4]   2> 296481 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 296501 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 296509 INFO  (zkCallback-402-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@79d9b8e6 name:ZooKeeperConnection Watcher:127.0.0.1:34436 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296509 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 296509 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 296510 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9750F61F0C7C897A

[...truncated too long message...]

bstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>   47) Thread[id=9146, name=searcherExecutor-3746-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>   48) Thread[id=9155, name=qtp168669593-9155, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
   [junit4]   2>         at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
   [junit4]   2>         at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
   [junit4]   2>         at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]   2>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2>         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
   [junit4]   2>         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
   [junit4]   2>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
   [junit4]   2>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]   2>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>   49) Thread[id=9265, name=Scheduler-1735382488, state=TIMED_WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>   50) Thread[id=9180, name=searcherExecutor-3757-thread-1, state=WAITING, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=ClassicSimilarity, locale=hr, timezone=America/Nassau
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=86,free=178455912,total=511705088
   [junit4]   2> NOTE: All tests run in this JVM: [TestCloudInspectUtil, HdfsNNFailoverTest, OpenCloseCoreStressTest, CoreAdminCreateDiscoverTest, HdfsTlogReplayBufferedWhileIndexingTest, TestCoreContainer, ZkControllerTest, SSLMigrationTest, FileUtilsTest, DistributedTermsComponentTest, TestExceedMaxTermLength, ZkSolrClientTest, HdfsChaosMonkeyNothingIsSafeTest, ZkStateWriterTest, TestSchemaSimilarityResource, TestBlobHandler, TestTolerantUpdateProcessorRandomCloud, TestShortCircuitedRequests, TimeZoneUtilsTest, TestSearcherReuse, ShardRoutingCustomTest, LoggingHandlerTest, SolrPluginUtilsTest, OverseerModifyCollectionTest, DirectSolrSpellCheckerTest, CursorMarkTest, TestFiltering, TestStressUserVersions, TestCursorMarkWithoutUniqueKey, HttpPartitionTest, TestRequestForwarding, TestCloudSchemaless, ClusterStateUpdateTest, BooleanFieldTest, CSVRequestHandlerTest, SmileWriterTest, CdcrReplicationHandlerTest, MBeansHandlerTest, SpellCheckCollatorTest, TestQuerySenderNoQuery, UniqFieldsUpdateProcessorFactoryTest, TestCryptoKeys, TestLockTree, TestHashQParserPlugin, CollectionsAPIDistributedZkTest, SoftAutoCommitTest, AliasIntegrationTest, TestExclusionRuleCollectionAccess, ConvertedLegacyTest, TestQueryTypes, TestUpdate, SolrInfoMBeanTest, SpellPossibilityIteratorTest, NotRequiredUniqueKeyTest, InfoHandlerTest, TestReplicationHandlerBackup, ConfigSetsAPITest, AsyncMigrateRouteKeyTest, TestUninvertingReader, TestFieldTypeResource, DistributedSuggestComponentTest, DistanceFunctionTest, DocValuesMissingTest, TestCollapseQParserPlugin, NumericFieldsTest, WordBreakSolrSpellCheckerTest, TestSimpleTrackingShardHandler, MoreLikeThisHandlerTest, TestQuerySenderListener, SpellCheckComponentTest, TestObjectReleaseTracker, RulesTest, SpellingQueryConverterTest, CollectionReloadTest, DistributedIntervalFacetingTest, TestDFISimilarityFactory, SuggesterFSTTest, TestFieldCollectionResource, DeleteReplicaTest, DistribCursorPagingTest, TestMissingGroups, TestFieldCacheSortRandom, TestTestInjection, TestAnalyzeInfixSuggestions, TestStressCloudBlindAtomicUpdates, SharedFSAutoReplicaFailoverTest, TestReRankQParserPlugin, TestQueryUtils, TestCodecSupport, OverseerTaskQueueTest, TestFieldSortValues, TestGraphMLResponseWriter, HdfsLockFactoryTest, TestInfoStreamLogging, DistributedQueueTest, TestPartialUpdateDeduplication, SortSpecParsingTest, BlockDirectoryTest, DeleteInactiveReplicaTest, LeaderInitiatedRecoveryOnCommitTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderInitiatedRecoveryOnCommitTest -Dtests.seed=9750F61F0C7C897A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=hr -Dtests.timezone=America/Nassau -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | LeaderInitiatedRecoveryOnCommitTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9750F61F0C7C897A]:0)
   [junit4] Completed [619/619 (4!)] on J1 in 7223.82s, 1 test, 2 errors <<< FAILURES!

[...truncated 15 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:740: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/common-build.xml:530: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/lucene/common-build.xml:1427: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/lucene/common-build.xml:984: There were test failures: 619 suites (10 ignored), 2597 tests, 1 suite-level error, 1 error, 3 failures, 87 ignored (72 assumptions) [seed: 9750F61F0C7C897A]

Total time: 174 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any