You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/07/27 07:09:40 UTC

[JENKINS-EA] Lucene-Solr-6.x-Linux (32bit/jdk-9-ea+127) - Build # 1285 - Still Unstable!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1285/
Java: 32bit/jdk-9-ea+127 -client -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.overseer.ZkStateWriterTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest:     1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest: 
   1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
	at __randomizedtesting.SeedInfo.seed([CC00B78135B21353]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.overseer.ZkStateWriterTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
        at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
        at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
	at __randomizedtesting.SeedInfo.seed([CC00B78135B21353]:0)




Build Log:
[...truncated 11137 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateWriterTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.overseer.ZkStateWriterTest_CC00B78135B21353-001/init-core-data-001
   [junit4]   2> 383793 INFO  (SUITE-ZkStateWriterTest-seed#[CC00B78135B21353]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 383795 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToSharedClusterState
   [junit4]   2> 383795 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 383795 INFO  (Thread-651) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 383795 INFO  (Thread-651) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 383895 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer start zk server on port:41201
   [junit4]   2> 383895 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 383896 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 383897 INFO  (zkCallback-524-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@17a298 name:ZooKeeperConnection Watcher:127.0.0.1:41201 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 383897 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 383898 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 383898 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 383898 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 383899 INFO  (zkCallback-525-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50ae45 name:ZooKeeperConnection Watcher:127.0.0.1:41201 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 383899 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 383899 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 383899 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 383902 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 383902 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 383902 INFO  (zkCallback-526-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@18487a name:ZooKeeperConnection Watcher:127.0.0.1:41201/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 383902 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 383903 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 383903 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 383903 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 383904 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 383904 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 383904 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 383905 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 383905 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 383905 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 383905 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 383906 INFO  (zkCallback-526-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 383906 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 383906 INFO  (zkCallback-526-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 383906 INFO  (zkCallback-526-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 383907 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c1
   [junit4]   2> 383907 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c2
   [junit4]   2> 383907 INFO  (zkCallback-526-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 383908 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41201 41201
   [junit4]   2> 388930 INFO  (Thread-651) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41201 41201
   [junit4]   2> 388930 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToSharedClusterState
   [junit4]   2> 388932 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSingleExternalCollection
   [junit4]   2> 388932 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 388932 INFO  (Thread-652) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 388932 INFO  (Thread-652) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 389032 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer start zk server on port:45108
   [junit4]   2> 389032 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 389033 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 389034 INFO  (zkCallback-528-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1f28e41 name:ZooKeeperConnection Watcher:127.0.0.1:45108 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389034 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 389035 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 389035 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 389035 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 389036 INFO  (zkCallback-529-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1d0caae name:ZooKeeperConnection Watcher:127.0.0.1:45108 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389036 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 389037 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 389037 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 389038 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 389038 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 389038 INFO  (zkCallback-530-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@170ea45 name:ZooKeeperConnection Watcher:127.0.0.1:45108/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389039 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 389039 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 389039 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 389039 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 389040 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 389041 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 389041 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 389042 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 389042 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 389042 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 389043 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 389043 INFO  (zkCallback-530-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 389044 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 389045 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45108 45108
   [junit4]   2> 394156 INFO  (Thread-652) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45108 45108
   [junit4]   2> 394157 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSingleExternalCollection
   [junit4]   2> 394159 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSingleLegacyCollection
   [junit4]   2> 394159 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 394159 INFO  (Thread-653) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 394159 INFO  (Thread-653) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 394259 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer start zk server on port:38748
   [junit4]   2> 394259 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 394260 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 394261 INFO  (zkCallback-532-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@9e4639 name:ZooKeeperConnection Watcher:127.0.0.1:38748 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 394261 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 394261 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 394262 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 394262 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 394262 INFO  (zkCallback-533-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1d35f86 name:ZooKeeperConnection Watcher:127.0.0.1:38748 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 394263 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 394263 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 394263 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 394264 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 394264 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 394265 INFO  (zkCallback-534-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1e48628 name:ZooKeeperConnection Watcher:127.0.0.1:38748/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 394265 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 394265 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 394266 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 394266 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 394267 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 394267 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 394268 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 394268 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 394268 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 394268 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 394269 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 394273 ERROR (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$149/24037428@1e113df rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1de897b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(java.base@9-ea/ThreadPoolExecutor.java:2073)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(java.base@9-ea/ThreadPoolExecutor.java:839)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(java.base@9-ea/ThreadPoolExecutor.java:1385)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(java.base@9-ea/AbstractExecutorService.java:118)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 394273 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38748 38748
   [junit4]   2> 394273 ERROR (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$149/24037428@a0830e rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1de897b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(java.base@9-ea/ThreadPoolExecutor.java:2073)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(java.base@9-ea/ThreadPoolExecutor.java:839)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(java.base@9-ea/ThreadPoolExecutor.java:1385)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(java.base@9-ea/AbstractExecutorService.java:118)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 399286 INFO  (Thread-653) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38748 38748
   [junit4]   2> 399287 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSingleLegacyCollection
   [junit4]   2> 399288 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToStateFormat2
   [junit4]   2> 399289 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 399289 INFO  (Thread-654) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 399289 INFO  (Thread-654) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 399389 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer start zk server on port:39863
   [junit4]   2> 399389 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 399390 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 399391 INFO  (zkCallback-536-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@17254e0 name:ZooKeeperConnection Watcher:127.0.0.1:39863 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 399391 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 399392 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 399392 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1562b18759c0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 399393 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 399394 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 399394 INFO  (zkCallback-537-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@173b5da name:ZooKeeperConnection Watcher:127.0.0.1:39863 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 399394 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 399395 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 399395 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 399406 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1562b18759c0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> 399406 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 399407 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 399408 INFO  (zkCallback-538-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3d4770 name:ZooKeeperConnection Watcher:127.0.0.1:39863/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 399408 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 399408 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 399408 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 399409 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 399410 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 399410 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 399411 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 399411 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 399412 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 399412 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 399413 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 399417 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 399418 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c2/state.json
   [junit4]   2> 399419 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Checking legacy cluster state for collection c2
   [junit4]   2> 399420 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Adding lazily-loaded reference for collection c2
   [junit4]   2> 399420 INFO  (zkCallback-538-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 399421 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.o.ZkStateWriterTest Cluster state: live nodes:[]collections:{c2=LazyCollectionRef(c2)}
   [junit4]   2> 399421 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c2
   [junit4]   2> 399422 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/c2/state.json version: 0
   [junit4]   2> 399423 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39863 39863
   [junit4]   2> 404456 INFO  (Thread-654) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39863 39863
   [junit4]   2> 404456 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToStateFormat2
   [junit4]   2> 404458 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZkStateWriterBatching
   [junit4]   2> 404458 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 404459 INFO  (Thread-655) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 404459 INFO  (Thread-655) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 404559 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer start zk server on port:39300
   [junit4]   2> 404559 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 404559 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 404560 INFO  (zkCallback-540-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1f74715 name:ZooKeeperConnection Watcher:127.0.0.1:39300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 404560 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 404560 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 404561 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 404561 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 404561 INFO  (zkCallback-541-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5db8e3 name:ZooKeeperConnection Watcher:127.0.0.1:39300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 404562 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 404562 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 404562 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 404563 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 404563 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 404563 INFO  (zkCallback-542-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@99230f name:ZooKeeperConnection Watcher:127.0.0.1:39300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 404563 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 404563 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 404564 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 404564 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 404564 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 404565 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 404565 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 404565 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 404566 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 404566 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 404566 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 404566 INFO  (zkCallback-542-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 404566 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 404567 INFO  (zkCallback-542-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 404567 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 404568 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39300 39300
   [junit4]   2> 404781 INFO  (Thread-655) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39300 39300
   [junit4]   2> 404781 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[CC00B78135B21353]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZkStateWriterBatching
   [junit4]   2> 404782 INFO  (SUITE-ZkStateWriterTest-seed#[CC00B78135B21353]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Jul 27, 2016 6:44:03 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Jul 27, 2016 6:44:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest: 
   [junit4]   2>    1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]   2>         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
   [junit4]   2>         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> Jul 27, 2016 6:44:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]   2> Jul 27, 2016 6:44:27 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]   2>         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
   [junit4]   2>         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=1845, sim=ClassicSimilarity, locale=jmc-TZ, timezone=America/Knox_IN
   [junit4]   2> NOTE: Linux 4.4.0-31-generic i386/Oracle Corporation 9-ea (32-bit)/cpus=12,threads=2,free=160666456,total=292814848
   [junit4]   2> NOTE: All tests run in this JVM: [MultiTermTest, TestSimpleTrackingShardHandler, PreAnalyzedUpdateProcessorTest, DOMUtilTest, DocExpirationUpdateProcessorFactoryTest, CollectionReloadTest, TestQuerySenderListener, TestWordDelimiterFilterFactory, TestCryptoKeys, StressHdfsTest, TestRTimerTree, TestTrie, SpellPossibilityIteratorTest, TestSurroundQueryParser, TestPKIAuthenticationPlugin, TestFastLRUCache, TestPartialUpdateDeduplication, SolrCLIZkUtilsTest, TestSolrConfigHandlerConcurrent, WordBreakSolrSpellCheckerTest, TestBlendedInfixSuggestions, TestDistributedStatsComponentCardinality, CollectionsAPIAsyncDistributedZkTest, ConfigSetsAPITest, MinimalSchemaTest, DistributedMLTComponentTest, TestMiniSolrCloudClusterSSL, SolrIndexSplitterTest, SpellCheckComponentTest, HdfsRestartWhileUpdatingTest, TestRestoreCore, TestLMJelinekMercerSimilarityFactory, SolrPluginUtilsTest, ExplicitHLLTest, ZkStateWriterTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateWriterTest -Dtests.seed=CC00B78135B21353 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=jmc-TZ -Dtests.timezone=America/Knox_IN -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | ZkStateWriterTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest: 
   [junit4]    >    1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]    >         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
   [junit4]    >         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CC00B78135B21353]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=2825, name=watches-539-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]    >         at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
   [junit4]    >         at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CC00B78135B21353]:0)
   [junit4] Completed [158/624 (1!)] on J1 in 44.05s, 5 tests, 2 errors <<< FAILURES!

[...truncated 51145 lines...]