You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2016/07/11 01:24:12 UTC

[JENKINS] Lucene-Solr-Tests-6.x - Build # 328 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/328/

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.overseer.ZkStateWriterTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest:     1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest: 
   1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([9EA0852D43F467EA]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.overseer.ZkStateWriterTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([9EA0852D43F467EA]:0)




Build Log:
[...truncated 10913 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateWriterTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateWriterTest_9EA0852D43F467EA-001/init-core-data-001
   [junit4]   2> 387460 INFO  (SUITE-ZkStateWriterTest-seed#[9EA0852D43F467EA]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 387463 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSingleLegacyCollection
   [junit4]   2> 387464 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 387464 INFO  (Thread-688) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 387464 INFO  (Thread-688) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 387564 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer start zk server on port:43353
   [junit4]   2> 387564 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 387573 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 387579 INFO  (zkCallback-336-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@24ab8ace name:ZooKeeperConnection Watcher:127.0.0.1:43353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 387579 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 387580 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 387581 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155d7810b030000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 387583 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 387583 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 387584 INFO  (zkCallback-337-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@a0a18e2 name:ZooKeeperConnection Watcher:127.0.0.1:43353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 387584 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 387584 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 387585 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 387588 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 387595 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 387602 INFO  (zkCallback-338-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1fce679b name:ZooKeeperConnection Watcher:127.0.0.1:43353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 387602 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 387603 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 387603 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 387604 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 387605 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 387606 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 387607 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 387608 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 387608 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 387609 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 387610 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 387611 INFO  (zkCallback-338-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 387626 INFO  (zkCallback-338-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 387626 WARN  (zkCallback-338-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /clusterstate.json]
   [junit4]   2> 387626 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43353 43353
   [junit4]   2> 387844 INFO  (Thread-688) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43353 43353
   [junit4]   2> 387845 INFO  (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSingleLegacyCollection
   [junit4]   2> 387852 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZkStateWriterBatching
   [junit4]   2> 387852 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 387852 INFO  (Thread-689) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 387852 INFO  (Thread-689) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 387952 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer start zk server on port:36012
   [junit4]   2> 387952 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 387953 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 387955 INFO  (zkCallback-340-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@d252c45 name:ZooKeeperConnection Watcher:127.0.0.1:36012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 387955 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 387955 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 387960 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 387960 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 387963 INFO  (zkCallback-341-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@555c560 name:ZooKeeperConnection Watcher:127.0.0.1:36012 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 387964 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 387964 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 387964 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 387969 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 387969 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 387971 INFO  (zkCallback-342-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@16c89200 name:ZooKeeperConnection Watcher:127.0.0.1:36012/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 387971 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 387971 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 387971 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 387972 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 387974 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 387975 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 387976 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 387976 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 387977 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 387978 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 387979 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 387980 INFO  (zkCallback-342-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 387980 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 387981 INFO  (zkCallback-342-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 387982 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 387986 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36012 36012
   [junit4]   2> 388085 INFO  (Thread-689) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36012 36012
   [junit4]   2> 388086 INFO  (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZkStateWriterBatching
   [junit4]   2> 388088 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSingleExternalCollection
   [junit4]   2> 388088 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 388099 INFO  (Thread-690) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 388099 INFO  (Thread-690) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 388198 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer start zk server on port:34801
   [junit4]   2> 388198 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388206 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388208 INFO  (zkCallback-344-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3981310e name:ZooKeeperConnection Watcher:127.0.0.1:34801 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388208 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388209 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388218 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388219 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388241 INFO  (zkCallback-345-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@438d35e6 name:ZooKeeperConnection Watcher:127.0.0.1:34801 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388241 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388242 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388242 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 388245 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388251 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388265 INFO  (zkCallback-346-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2781f2e2 name:ZooKeeperConnection Watcher:127.0.0.1:34801/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388265 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388265 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388266 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 388267 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 388267 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 388268 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 388269 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 388270 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 388270 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 388271 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 388272 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 388272 INFO  (zkCallback-346-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 388273 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 388281 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34801 34801
   [junit4]   2> 388445 INFO  (Thread-690) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34801 34801
   [junit4]   2> 388446 INFO  (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSingleExternalCollection
   [junit4]   2> 388448 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToSharedClusterState
   [junit4]   2> 388448 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 388448 INFO  (Thread-691) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 388448 INFO  (Thread-691) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 388548 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer start zk server on port:54028
   [junit4]   2> 388548 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388549 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388553 INFO  (zkCallback-348-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@e8cb254 name:ZooKeeperConnection Watcher:127.0.0.1:54028 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388553 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388553 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388554 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388555 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388556 INFO  (zkCallback-349-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7daf9030 name:ZooKeeperConnection Watcher:127.0.0.1:54028 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388556 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388556 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388556 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 388559 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388559 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388567 INFO  (zkCallback-350-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@a5ba54d name:ZooKeeperConnection Watcher:127.0.0.1:54028/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388567 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388567 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388568 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 388569 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 388570 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 388571 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 388579 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 388580 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 388580 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 388581 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 388582 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 388583 INFO  (zkCallback-350-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 388583 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 388592 INFO  (zkCallback-350-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 388592 INFO  (zkCallback-350-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 388595 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Checking legacy cluster state for collection c1
   [junit4]   2> 388595 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c2
   [junit4]   2> 388596 INFO  (zkCallback-350-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 388602 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54028 54028
   [junit4]   2> 388758 INFO  (Thread-691) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54028 54028
   [junit4]   2> 388759 INFO  (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToSharedClusterState
   [junit4]   2> 388761 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToStateFormat2
   [junit4]   2> 388762 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 388762 INFO  (Thread-692) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 388762 INFO  (Thread-692) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 388862 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer start zk server on port:57267
   [junit4]   2> 388863 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388863 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388867 INFO  (zkCallback-352-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5487c823 name:ZooKeeperConnection Watcher:127.0.0.1:57267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388867 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388868 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388869 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388871 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388877 INFO  (zkCallback-353-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@362e7711 name:ZooKeeperConnection Watcher:127.0.0.1:57267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388877 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388877 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388877 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 388879 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 388883 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 388884 INFO  (zkCallback-354-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@9ae9c3d name:ZooKeeperConnection Watcher:127.0.0.1:57267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 388884 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 388885 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 388885 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 388886 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 388887 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 388888 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 388889 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 388890 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 388891 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 388891 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 388892 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 388893 INFO  (zkCallback-354-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 388894 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
   [junit4]   2> 388895 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c2/state.json
   [junit4]   2> 388896 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Checking legacy cluster state for collection c2
   [junit4]   2> 388896 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Adding lazily-loaded reference for collection c2
   [junit4]   2> 388899 INFO  (zkCallback-354-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 388903 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.o.ZkStateWriterTest Cluster state: live nodes:[]collections:{c1=LazyCollectionRef(c1), c2=LazyCollectionRef(c2)}
   [junit4]   2> 388904 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c2
   [junit4]   2> 388905 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/c2/state.json version: 0
   [junit4]   2> 388906 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57267 57267
   [junit4]   2> 389079 INFO  (Thread-692) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57267 57267
   [junit4]   2> 389080 INFO  (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[9EA0852D43F467EA]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToStateFormat2
   [junit4]   2> 389081 INFO  (SUITE-ZkStateWriterTest-seed#[9EA0852D43F467EA]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Jul 11, 2016 1:10:07 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> Jul 11, 2016 1:10:28 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest: 
   [junit4]   2>    1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Jul 11, 2016 1:10:28 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]   2> Jul 11, 2016 1:10:31 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]   2>         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=583, maxMBSortInHeap=6.333189609129963, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=ar-LB, timezone=Pacific/Niue
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=223870520,total=522715136
   [junit4]   2> NOTE: All tests run in this JVM: [UUIDFieldTest, TestConfigSets, TestCollapseQParserPlugin, ExternalFileFieldSortTest, TestSubQueryTransformer, TestCloudInspectUtil, DistribJoinFromCollectionTest, TestFreeTextSuggestions, StandardRequestHandlerTest, OutputWriterTest, TestSolrDynamicMBean, TestBadConfig, SimpleCollectionCreateDeleteTest, TestOmitPositions, TestRandomFaceting, SystemInfoHandlerTest, SignatureUpdateProcessorFactoryTest, TestFieldCacheReopen, URLClassifyProcessorTest, ResourceLoaderTest, BlockDirectoryTest, ZkNodePropsTest, TestMergePolicyConfig, LeaderInitiatedRecoveryOnCommitTest, TestDefaultSearchFieldResource, TestRandomRequestDistribution, XsltUpdateRequestHandlerTest, TestCollectionAPI, QueryElevationComponentTest, DistributedFacetPivotLongTailTest, TestSolrQueryParserDefaultOperatorResource, JsonLoaderTest, TestRangeQuery, MultiThreadedOCPTest, ZkStateWriterTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateWriterTest -Dtests.seed=9EA0852D43F467EA -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-LB -Dtests.timezone=Pacific/Niue -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | ZkStateWriterTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest: 
   [junit4]    >    1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9EA0852D43F467EA]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=2190, name=watches-355-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9EA0852D43F467EA]:0)
   [junit4] Completed [96/621 (1!)] on J0 in 24.73s, 5 tests, 2 errors <<< FAILURES!

[...truncated 1681 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:740: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/common-build.xml:530: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/lucene/common-build.xml:984: There were test failures: 621 suites (10 ignored), 2642 tests, 2 suite-level errors, 1607 ignored (73 assumptions) [seed: 9EA0852D43F467EA]

Total time: 53 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any