You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/08/01 19:03:14 UTC

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 754 - Still Unstable!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/754/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
Could not find collection:test_col

Stack Trace:
java.lang.AssertionError: Could not find collection:test_col
	at __randomizedtesting.SeedInfo.seed([C81822661B323C6D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorRandomCloud.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor]
	at __randomizedtesting.SeedInfo.seed([C81822661B323C6D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores:     1) Thread[id=11622, name=searcherExecutor-4887-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 
   1) Thread[id=11622, name=searcherExecutor-4887-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([C81822661B323C6D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=11622, name=searcherExecutor-4887-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=11622, name=searcherExecutor-4887-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([C81822661B323C6D]:0)


FAILED:  org.apache.solr.common.cloud.TestCollectionStateWatchers.testWaitForStateChecksCurrentState

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([50D623F7CC8F6B7E:863A4C2F9CD6162B]:0)
	at org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1254)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:651)
	at org.apache.solr.common.cloud.TestCollectionStateWatchers.testWaitForStateChecksCurrentState(TestCollectionStateWatchers.java:158)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11400 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/init-core-data-001
   [junit4]   2> 1340851 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-9182 - causes OOM)
   [junit4]   2> 1340851 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.TestTolerantUpdateProcessorRandomCloud Configuring cluster: servers=7, shards=2, repfactor=3
   [junit4]   2> 1340852 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1340852 INFO  (Thread-2991) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1340852 INFO  (Thread-2991) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1340952 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:39709
   [junit4]   2> 1340952 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1340952 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1340955 INFO  (zkCallback-1650-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c060fcf name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1340955 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1340955 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1340955 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[C81822661B323C6D]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 1340960 INFO  (jetty-launcher-1649-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1340960 INFO  (jetty-launcher-1649-thread-2) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1340962 INFO  (jetty-launcher-1649-thread-3) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1340963 INFO  (jetty-launcher-1649-thread-4) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1340965 INFO  (jetty-launcher-1649-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4009e6ac{/solr,null,AVAILABLE}
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@67078b6d{HTTP/1.1,[http/1.1]}{127.0.0.1:39387}
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.e.j.s.Server Started @1344724ms
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39387}
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node2'
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1340966 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1340967 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1340968 INFO  (jetty-launcher-1649-thread-7) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1340969 INFO  (jetty-launcher-1649-thread-5) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1340970 INFO  (jetty-launcher-1649-thread-6) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1340972 INFO  (jetty-launcher-1649-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@59cf036d{/solr,null,AVAILABLE}
   [junit4]   2> 1340972 INFO  (jetty-launcher-1649-thread-6) [    ] o.e.j.s.ServerConnector Started ServerConnector@b6fd935{HTTP/1.1,[http/1.1]}{127.0.0.1:64534}
   [junit4]   2> 1340972 INFO  (jetty-launcher-1649-thread-6) [    ] o.e.j.s.Server Started @1344730ms
   [junit4]   2> 1340972 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64534}
   [junit4]   2> 1340972 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1340972 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node7'
   [junit4]   2> 1340973 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1340973 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1340973 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1340973 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1340975 INFO  (zkCallback-1652-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4bd54d86 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1340975 INFO  (jetty-launcher-1649-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@21cff82c{/solr,null,AVAILABLE}
   [junit4]   2> 1340975 INFO  (jetty-launcher-1649-thread-4) [    ] o.e.j.s.ServerConnector Started ServerConnector@2130bf89{HTTP/1.1,[http/1.1]}{127.0.0.1:52348}
   [junit4]   2> 1340975 INFO  (jetty-launcher-1649-thread-4) [    ] o.e.j.s.Server Started @1344733ms
   [junit4]   2> 1340975 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52348}
   [junit4]   2> 1340976 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1340976 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node4'
   [junit4]   2> 1340976 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1340976 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1340976 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1340976 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1340977 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1340978 INFO  (zkCallback-1653-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@197025b4 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1340978 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1340978 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1340978 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1340978 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1340987 INFO  (jetty-launcher-1649-thread-7) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@642ee366{/solr,null,AVAILABLE}
   [junit4]   2> 1340987 INFO  (jetty-launcher-1649-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@57df8141{/solr,null,AVAILABLE}
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.e.j.s.ServerConnector Started ServerConnector@7d3ffc4b{HTTP/1.1,[http/1.1]}{127.0.0.1:61463}
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.e.j.s.Server Started @1344745ms
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61463}
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@636c9b9e{HTTP/1.1,[http/1.1]}{127.0.0.1:64384}
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.e.j.s.Server Started @1344746ms
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64384}
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node6'
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node1'
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1340988 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1340989 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1340990 INFO  (zkCallback-1651-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1adbf594 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1340991 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1340991 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1340991 INFO  (zkCallback-1654-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5707b8e0 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1340992 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1340992 INFO  (jetty-launcher-1649-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@226e9875{/solr,null,AVAILABLE}
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@7102f42c{HTTP/1.1,[http/1.1]}{127.0.0.1:43153}
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.e.j.s.Server Started @1344751ms
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43153}
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node3'
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1340993 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1340994 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1340994 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1340995 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1340995 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1340995 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1340998 INFO  (jetty-launcher-1649-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@48a88cce{/solr,null,AVAILABLE}
   [junit4]   2> 1340998 INFO  (jetty-launcher-1649-thread-5) [    ] o.e.j.s.ServerConnector Started ServerConnector@4bba517e{HTTP/1.1,[http/1.1]}{127.0.0.1:36509}
   [junit4]   2> 1340999 INFO  (jetty-launcher-1649-thread-5) [    ] o.e.j.s.Server Started @1344756ms
   [junit4]   2> 1340999 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36509}
   [junit4]   2> 1340999 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1340999 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node5'
   [junit4]   2> 1340999 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1340999 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1341001 INFO  (zkCallback-1655-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4610594b name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341001 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341002 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341002 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1341006 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node1/.
   [junit4]   2> 1341006 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1341006 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 685987041
   [junit4]   2> 1341006 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node1]
   [junit4]   2> 1341006 WARN  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node1/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node1/lib
   [junit4]   2> 1341006 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341007 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1341007 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341008 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 1341008 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341008 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341008 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39709/solr
   [junit4]   2> 1341008 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341008 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341009 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341009 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node4/.
   [junit4]   2> 1341009 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.CoreContainer New CoreContainer 473572679
   [junit4]   2> 1341009 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node4]
   [junit4]   2> 1341009 WARN  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node4/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node4/lib
   [junit4]   2> 1341010 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39709/solr
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node7/.
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.CoreContainer New CoreContainer 1332839554
   [junit4]   2> 1341011 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node7]
   [junit4]   2> 1341012 WARN  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node7/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node7/lib
   [junit4]   2> 1341012 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341012 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node2/.
   [junit4]   2> 1341012 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1341012 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.CoreContainer New CoreContainer 353149103
   [junit4]   2> 1341012 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node2]
   [junit4]   2> 1341012 WARN  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node2/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node2/lib
   [junit4]   2> 1341012 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1341013 INFO  (zkCallback-1656-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5ab12517 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341013 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341013 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341013 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 1341013 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341013 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341013 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39709/solr
   [junit4]   2> 1341013 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39709/solr
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node6/.
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.CoreContainer New CoreContainer 287645737
   [junit4]   2> 1341014 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node6]
   [junit4]   2> 1341015 WARN  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node6/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node6/lib
   [junit4]   2> 1341015 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1341016 INFO  (zkCallback-1669-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7fc93316 name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39709/solr
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341016 INFO  (zkCallback-1660-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4a82edf7 name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341016 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341017 INFO  (jetty-launcher-1649-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341017 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1341018 INFO  (zkCallback-1657-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d5b399b name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341018 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341018 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$73/567155726@7fed16b7
   [junit4]   2> 1341018 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341019 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341019 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1341020 INFO  (zkCallback-1663-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@b224c57 name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341021 INFO  (zkCallback-1673-thread-1-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1433a2ee name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341021 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341021 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$73/567155726@75b48a8
   [junit4]   2> 1341022 INFO  (zkCallback-1672-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4029c490 name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341022 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341022 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341022 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341022 INFO  (jetty-launcher-1649-thread-7) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341022 INFO  (zkCallback-1668-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@65317177 name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341022 INFO  (jetty-launcher-1649-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341023 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1341024 INFO  (zkCallback-1674-thread-1-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c08974b name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341024 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341024 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$73/567155726@11bd90fb
   [junit4]   2> 1341024 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341025 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$73/567155726@6f646064
   [junit4]   2> 1341025 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341026 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341026 INFO  (jetty-launcher-1649-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341026 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node3/.
   [junit4]   2> 1341026 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@32d1d624 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341026 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.CoreContainer New CoreContainer 1926547474
   [junit4]   2> 1341026 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node3]
   [junit4]   2> 1341026 INFO  (zkCallback-1676-thread-1-processing-n:127.0.0.1:52348_solr) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@72e6aa20 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341026 WARN  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node3/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node3/lib
   [junit4]   2> 1341026 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341027 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1341027 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1341028 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 1341028 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341028 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341028 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39709/solr
   [junit4]   2> 1341028 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341028 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341029 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341029 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341030 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1564766f61c0007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1341030 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node5/.
   [junit4]   2> 1341030 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.CoreContainer New CoreContainer 1766386258
   [junit4]   2> 1341030 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node5]
   [junit4]   2> 1341030 WARN  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node5/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node5/lib
   [junit4]   2> 1341031 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1341031 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1564766f61c0009, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1341033 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1341033 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 1341033 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341033 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341033 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39709/solr
   [junit4]   2> 1341033 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341033 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341034 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341036 INFO  (zkCallback-1679-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3b134b2 name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341036 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$73/567155726@12f7933f
   [junit4]   2> 1341036 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341036 INFO  (jetty-launcher-1649-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341036 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1341036 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341037 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1341038 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$73/567155726@42837a8c
   [junit4]   2> 1341038 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341038 INFO  (zkCallback-1682-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7461ce1c name:ZooKeeperConnection Watcher:127.0.0.1:39709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341039 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1341039 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341039 INFO  (jetty-launcher-1649-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341040 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1341040 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6e3c619e name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341040 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1341040 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341041 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1341041 INFO  (zkCallback-1684-thread-1-processing-n:127.0.0.1:43153_solr) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@11f3c491 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341042 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341043 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1341043 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1341044 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1341045 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1341045 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1341046 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1341047 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1341048 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1341048 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1341050 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1341050 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1341051 INFO  (jetty-launcher-1649-thread-5) [n:127.0.0.1:36509_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$73/567155726@6211b247
   [junit4]   2> 1341051 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341051 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341052 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341052 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1341053 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1341053 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341054 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1341055 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1341056 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1341058 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341058 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1341058 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1341059 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1341059 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1341060 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341060 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341061 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96342949576310798-127.0.0.1:39387_solr-n_0000000000
   [junit4]   2> 1341061 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1341061 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1341063 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1341063 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39387_solr
   [junit4]   2> 1341063 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1341064 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341066 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341066 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96342949576310796-127.0.0.1:64384_solr-n_0000000001
   [junit4]   2> 1341066 INFO  (jetty-launcher-1649-thread-5) [n:127.0.0.1:36509_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341067 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341067 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1341067 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96342949576310804-127.0.0.1:43153_solr-n_0000000003
   [junit4]   2> 1341067 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96342949576310798-127.0.0.1:39387_solr-n_0000000000 to know if I could be the leader
   [junit4]   2> 1341068 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64384_solr
   [junit4]   2> 1341068 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64384_solr
   [junit4]   2> 1341068 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96342949576310800-127.0.0.1:61463_solr-n_0000000002 to know if I could be the leader
   [junit4]   2> 1341067 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.Overseer Overseer (id=96342949576310798-127.0.0.1:39387_solr-n_0000000000) starting
   [junit4]   2> 1341066 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96342949576310800-127.0.0.1:61463_solr-n_0000000002
   [junit4]   2> 1341069 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43153_solr
   [junit4]   2> 1341069 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:43153_solr
   [junit4]   2> 1341070 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1341070 INFO  (zkCallback-1674-thread-1-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1341070 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1341070 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341071 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1341072 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1341072 INFO  (zkCallback-1684-thread-1-processing-n:127.0.0.1:43153_solr) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1341072 INFO  (zkCallback-1673-thread-1-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1341073 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1341073 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96342949576310796-127.0.0.1:64384_solr-n_0000000001 to know if I could be the leader
   [junit4]   2> 1341074 INFO  (zkCallback-1674-thread-1-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1341075 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96342949576310803-127.0.0.1:64534_solr-n_0000000004
   [junit4]   2> 1341076 INFO  (zkCallback-1684-thread-1-processing-n:127.0.0.1:43153_solr) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1341076 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:61463_solr as DOWN
   [junit4]   2> 1341076 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96342949576310804-127.0.0.1:43153_solr-n_0000000003 to know if I could be the leader
   [junit4]   2> 1341077 INFO  (zkCallback-1673-thread-2-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1341077 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:64534_solr as DOWN
   [junit4]   2> 1341078 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61463_solr
   [junit4]   2> 1341078 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:61463_solr
   [junit4]   2> 1341078 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64534_solr
   [junit4]   2> 1341078 INFO  (jetty-launcher-1649-thread-6) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64534_solr
   [junit4]   2> 1341079 INFO  (zkCallback-1673-thread-1-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1341079 INFO  (zkCallback-1684-thread-1-processing-n:127.0.0.1:43153_solr) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1341079 INFO  (zkCallback-1674-thread-1-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1341080 INFO  (zkCallback-1676-thread-1-processing-n:127.0.0.1:52348_solr) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1341080 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1341080 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1341081 INFO  (zkCallback-1684-thread-1-processing-n:127.0.0.1:43153_solr) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 1341089 INFO  (zkCallback-1673-thread-2-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 1341090 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 1341091 INFO  (zkCallback-1689-thread-1-processing-n:127.0.0.1:36509_solr) [n:127.0.0.1:36509_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@160e1e21 name:ZooKeeperConnection Watcher:127.0.0.1:39709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341091 INFO  (zkCallback-1674-thread-1-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 1341091 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 1341092 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1341094 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1341094 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1341095 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node1/.
   [junit4]   2> 1341096 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1341104 INFO  (jetty-launcher-1649-thread-5) [n:127.0.0.1:36509_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341107 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1341107 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1341108 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node6/.
   [junit4]   2> 1341108 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1341109 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1341109 INFO  (jetty-launcher-1649-thread-1) [n:127.0.0.1:64384_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1341110 INFO  (zkCallback-1676-thread-1-processing-n:127.0.0.1:52348_solr) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (4)
   [junit4]   2> 1341111 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341112 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96342949576310799-127.0.0.1:52348_solr-n_0000000005
   [junit4]   2> 1341112 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96342949576310803-127.0.0.1:64534_solr-n_0000000004 to know if I could be the leader
   [junit4]   2> 1341113 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:52348_solr as DOWN
   [junit4]   2> 1341114 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52348_solr
   [junit4]   2> 1341114 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:52348_solr
   [junit4]   2> 1341114 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1341114 INFO  (jetty-launcher-1649-thread-7) [n:127.0.0.1:61463_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1341117 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1341118 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1341118 INFO  (zkCallback-1674-thread-1-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1341118 INFO  (zkCallback-1673-thread-2-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1341118 INFO  (zkCallback-1676-thread-1-processing-n:127.0.0.1:52348_solr) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1341119 INFO  (zkCallback-1684-thread-1-processing-n:127.0.0.1:43153_solr) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1341120 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1341120 INFO  (zkCallback-1673-thread-2-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1341121 INFO  (zkCallback-1684-thread-1-processing-n:127.0.0.1:43153_solr) [n:127.0.0.1:43153_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1341121 INFO  (zkCallback-1674-thread-1-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1341121 INFO  (zkCallback-1676-thread-1-processing-n:127.0.0.1:52348_solr) [n:127.0.0.1:52348_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1341122 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 1341129 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1341129 INFO  (OverseerCollectionConfigSetProcessor-96342949576310798-127.0.0.1:39387_solr-n_0000000000) [n:127.0.0.1:39387_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1341132 INFO  (OverseerStateUpdate-96342949576310798-127.0.0.1:39387_solr-n_0000000000) [n:127.0.0.1:39387_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1341133 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:39387_solr as DOWN
   [junit4]   2> 1341134 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1341134 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1341135 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node3/.
   [junit4]   2> 1341135 INFO  (jetty-launcher-1649-thread-3) [n:127.0.0.1:43153_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1341136 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1341136 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1341139 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTolerantUpdateProcessorRandomCloud_C81822661B323C6D-001/tempDir-001/node4/.
   [junit4]   2> 1341139 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1341140 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1341140 INFO  (jetty-launcher-1649-thread-4) [n:127.0.0.1:52348_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1341144 INFO  (OverseerStateUpdate-96342949576310798-127.0.0.1:39387_solr-n_0000000000) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (5)
   [junit4]   2> 1341145 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39387_solr
   [junit4]   2> 1341145 INFO  (jetty-launcher-1649-thread-2) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:39387_solr
   [junit4]   2> 1341147 INFO  (zkCallback-1673-thread-2-processing-n:127.0.0.1:64384_solr) [n:127.0.0.1:64384_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1341147 INFO  (zkCallback-1675-thread-1-processing-n:127.0.0.1:61463_solr) [n:127.0.0.1:61463_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1341147 INFO  (zkCallback-1683-thread-1-processing-n:127.0.0.1:64534_solr) [n:127.0.0.1:64534_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [5])
   [junit4]   2> 1341148 INFO  (zkCallback-1676-thread-1-processin

[...truncated too long message...]

]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 105970 ERROR (TEST-TestCollectionStateWatchers.testWaitForStateWatcherIsRetainedOnPredicateFailure-seed#[50D623F7CC8F6B7E]-EventThread) [n:127.0.0.1:34224_solr    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$30/1762306947@2c489512 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1d9fe86f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 105970 INFO  (jetty-launcher-44-thread-6) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ef28079{/solr,null,UNAVAILABLE}
   [junit4]   2> 105973 INFO  (zkCallback-90-thread-1-processing-n:127.0.0.1:36322_solr) [n:127.0.0.1:36322_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 105972 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x156476c7acf0017, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 105984 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x156476c7acf001a, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 105987 WARN  (zkCallback-90-thread-2-processing-n:127.0.0.1:36322_solr) [n:127.0.0.1:36322_solr    ] o.a.s.c.LeaderElector 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:105)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 105987 INFO  (zkCallback-64-thread-1-processing-n:127.0.0.1:47473_solr) [n:127.0.0.1:47473_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 105987 WARN  (zkCallback-64-thread-1-processing-n:127.0.0.1:47473_solr) [n:127.0.0.1:47473_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 105987 INFO  (jetty-launcher-44-thread-5) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@35c991c0{/solr,null,UNAVAILABLE}
   [junit4]   2> 107474 WARN  (zkCallback-90-thread-1-processing-n:127.0.0.1:36322_solr) [n:127.0.0.1:36322_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 107475 INFO  (jetty-launcher-44-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@26da3824{/solr,null,UNAVAILABLE}
   [junit4]   2> 107481 INFO  (SUITE-TestCollectionStateWatchers-seed#[50D623F7CC8F6B7E]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64706 64706
   [junit4]   2> 107508 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
   [junit4]   2> java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 107510 INFO  (Thread-31) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64706 64706
   [junit4]   2> 107526 WARN  (Thread-31) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/aliases.json
   [junit4]   2> 	7	/solr/configs/config
   [junit4]   2> 	7	/solr/clusterprops.json
   [junit4]   2> 	6	/solr/security.json
   [junit4]   2> 	3	/solr/collections/stateformat1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/collections/falsepredicate/state.json
   [junit4]   2> 	7	/solr/clusterstate.json
   [junit4]   2> 	6	/solr/collections/testcollection/state.json
   [junit4]   2> 	3	/solr/collections/stateformat1/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/96342973277470730-127.0.0.1:40588_solr-n_0000000000
   [junit4]   2> 	2	/solr/collections/currentstate/state.json
   [junit4]   2> 	2	/solr/collections/delayed/state.json
   [junit4]   2> 	2	/solr/collections/tobedeleted/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/96342973277470731-127.0.0.1:47473_solr-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	66	/solr/overseer/queue
   [junit4]   2> 	39	/solr/overseer/collection-queue-work
   [junit4]   2> 	19	/solr/overseer/queue-work
   [junit4]   2> 	7	/solr/collections
   [junit4]   2> 	5	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 107527 INFO  (SUITE-TestCollectionStateWatchers-seed#[50D623F7CC8F6B7E]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_50D623F7CC8F6B7E-001
   [junit4]   2> Aug 01, 2016 6:46:12 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=174, maxMBSortInHeap=5.62183115852416, sim=ClassicSimilarity, locale=el-CY, timezone=America/Scoresbysund
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=3,threads=1,free=35142024,total=120365056
   [junit4]   2> NOTE: All tests run in this JVM: [SolrExampleXMLTest, TestUpdateRequestCodec, SolrExampleEmbeddedTest, JdbcDriverTest, SolrExampleStreamingTest, ConcatOperationTest, TestXMLEscaping, CommonAdminParamsTest, ShardParamsTest, ModifiableSolrParamsTest, TestCollectionStateWatchers]
   [junit4] Completed [53/85 (1!)] on J0 in 66.35s, 9 tests, 1 error <<< FAILURES!

[...truncated 59508 lines...]