You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/07/24 19:54:08 UTC
[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+127) - Build
# 17354 - Still Unstable!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17354/
Java: 32bit/jdk-9-ea+127 -server -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.overseer.ZkStateWriterTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest: 1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest] at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632) at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest:
1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
at __randomizedtesting.SeedInfo.seed([51A4D395291ABC69]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.overseer.ZkStateWriterTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest] at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937) at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632) at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
at __randomizedtesting.SeedInfo.seed([51A4D395291ABC69]:0)
Build Log:
[...truncated 11460 lines...]
[junit4] Suite: org.apache.solr.cloud.overseer.ZkStateWriterTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.overseer.ZkStateWriterTest_51A4D395291ABC69-001/init-core-data-001
[junit4] 2> 920822 INFO (SUITE-ZkStateWriterTest-seed#[51A4D395291ABC69]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 920823 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSingleExternalCollection
[junit4] 2> 920823 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 920823 INFO (Thread-1320) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 920823 INFO (Thread-1320) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 920923 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer start zk server on port:43566
[junit4] 2> 920923 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 920923 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 920926 INFO (zkCallback-1054-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1628acd name:ZooKeeperConnection Watcher:127.0.0.1:43566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 920927 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 920927 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 920927 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 920928 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 920928 INFO (zkCallback-1055-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@185a95f name:ZooKeeperConnection Watcher:127.0.0.1:43566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 920928 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 920929 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 920929 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 920930 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 920931 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 920931 INFO (zkCallback-1056-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1f38552 name:ZooKeeperConnection Watcher:127.0.0.1:43566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 920932 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 920932 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 920932 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 920933 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 920933 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 920934 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 920934 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 920934 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 920935 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
[junit4] 2> 920935 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
[junit4] 2> 920937 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
[junit4] 2> 920937 INFO (zkCallback-1056-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
[junit4] 2> 920938 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
[junit4] 2> 920939 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43566 43566
[junit4] 2> 920966 INFO (Thread-1320) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43566 43566
[junit4] 2> 920967 INFO (TEST-ZkStateWriterTest.testSingleExternalCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSingleExternalCollection
[junit4] 2> 920993 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToStateFormat2
[junit4] 2> 920993 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 920993 INFO (Thread-1321) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 920993 INFO (Thread-1321) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 921093 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer start zk server on port:36141
[junit4] 2> 921093 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921094 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921095 INFO (zkCallback-1058-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@bc2313 name:ZooKeeperConnection Watcher:127.0.0.1:36141 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921096 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921096 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921096 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921097 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921097 INFO (zkCallback-1059-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3304f5 name:ZooKeeperConnection Watcher:127.0.0.1:36141 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921097 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921098 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921098 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 921099 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921099 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1561e6f84de0001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
[junit4] 2> 921099 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921100 INFO (zkCallback-1060-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@88d25b name:ZooKeeperConnection Watcher:127.0.0.1:36141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921100 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921100 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921101 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 921102 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 921103 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 921104 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 921111 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 921112 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 921112 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
[junit4] 2> 921112 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
[junit4] 2> 921113 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
[junit4] 2> 921113 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
[junit4] 2> 921114 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c2/state.json
[junit4] 2> 921114 INFO (zkCallback-1060-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
[junit4] 2> 921114 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Checking legacy cluster state for collection c2
[junit4] 2> 921116 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Adding lazily-loaded reference for collection c2
[junit4] 2> 921117 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.o.ZkStateWriterTest Cluster state: live nodes:[]collections:{c1=LazyCollectionRef(c1), c2=LazyCollectionRef(c2)}
[junit4] 2> 921117 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c2
[junit4] 2> 921118 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/c2/state.json version: 0
[junit4] 2> 921119 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36141 36141
[junit4] 2> 921230 INFO (Thread-1321) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36141 36141
[junit4] 2> 921231 INFO (TEST-ZkStateWriterTest.testExternalModificationToStateFormat2-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToStateFormat2
[junit4] 2> 921233 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testZkStateWriterBatching
[junit4] 2> 921233 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 921233 INFO (Thread-1322) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 921233 INFO (Thread-1322) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 921333 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer start zk server on port:38144
[junit4] 2> 921334 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921334 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921335 INFO (zkCallback-1062-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1420d36 name:ZooKeeperConnection Watcher:127.0.0.1:38144 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921335 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921336 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921336 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921336 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1561e6f85ce0000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
[junit4] 2> 921337 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921337 INFO (zkCallback-1063-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@358b3f name:ZooKeeperConnection Watcher:127.0.0.1:38144 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921337 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921338 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921338 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 921339 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1561e6f85ce0001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
[junit4] 2> 921339 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921339 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921340 INFO (zkCallback-1064-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@a30262 name:ZooKeeperConnection Watcher:127.0.0.1:38144/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921340 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921340 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921340 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 921341 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 921343 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 921343 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 921345 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 921345 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 921346 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
[junit4] 2> 921347 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
[junit4] 2> 921347 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
[junit4] 2> 921348 INFO (zkCallback-1064-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
[junit4] 2> 921348 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
[junit4] 2> 921348 INFO (zkCallback-1064-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
[junit4] 2> 921349 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
[junit4] 2> 921352 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38144 38144
[junit4] 2> 921369 INFO (Thread-1322) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38144 38144
[junit4] 2> 921370 INFO (TEST-ZkStateWriterTest.testZkStateWriterBatching-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testZkStateWriterBatching
[junit4] 2> 921372 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSingleLegacyCollection
[junit4] 2> 921373 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 921373 INFO (Thread-1323) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 921373 INFO (Thread-1323) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 921473 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer start zk server on port:40699
[junit4] 2> 921473 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921474 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921475 INFO (zkCallback-1066-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@192f64e name:ZooKeeperConnection Watcher:127.0.0.1:40699 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921475 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921475 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921476 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921477 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921477 INFO (zkCallback-1067-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@e64c7e name:ZooKeeperConnection Watcher:127.0.0.1:40699 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921478 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921478 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921478 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 921479 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921479 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921480 INFO (zkCallback-1068-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@102dbb5 name:ZooKeeperConnection Watcher:127.0.0.1:40699/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921480 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921480 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921480 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 921481 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 921481 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 921482 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 921485 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 921485 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 921485 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
[junit4] 2> 921486 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
[junit4] 2> 921486 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
[junit4] 2> 921487 INFO (zkCallback-1068-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
[junit4] 2> 921487 INFO (zkCallback-1068-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
[junit4] 2> 921489 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40699 40699
[junit4] 2> 921503 INFO (Thread-1323) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40699 40699
[junit4] 2> 921503 INFO (TEST-ZkStateWriterTest.testSingleLegacyCollection-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSingleLegacyCollection
[junit4] 2> 921506 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testExternalModificationToSharedClusterState
[junit4] 2> 921506 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 921506 INFO (Thread-1324) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 921506 INFO (Thread-1324) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 921606 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer start zk server on port:42817
[junit4] 2> 921606 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921607 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921609 INFO (zkCallback-1070-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@112d58 name:ZooKeeperConnection Watcher:127.0.0.1:42817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921609 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921610 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921611 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921612 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921612 INFO (zkCallback-1071-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1bb25fa name:ZooKeeperConnection Watcher:127.0.0.1:42817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921612 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921613 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921613 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 921616 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 921616 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 921617 INFO (zkCallback-1072-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@12974a6 name:ZooKeeperConnection Watcher:127.0.0.1:42817/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 921617 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 921617 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 921617 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
[junit4] 2> 921618 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections
[junit4] 2> 921618 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
[junit4] 2> 921619 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
[junit4] 2> 921619 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /security.json
[junit4] 2> 921628 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
[junit4] 2> 921628 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
[junit4] 2> 921629 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
[junit4] 2> 921630 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
[junit4] 2> 921631 INFO (zkCallback-1072-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
[junit4] 2> 921631 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.SolrZkClient makePath: /collections/c2
[junit4] 2> 921634 INFO (zkCallback-1072-thread-1) [ ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
[junit4] 2> 921635 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c1
[junit4] 2> 921635 INFO (zkCallback-1072-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
[junit4] 2> 921646 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.c.ZkStateReader Refreshing lazily-loaded state for collection c2
[junit4] 2> 921647 INFO (zkCallback-1072-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
[junit4] 2> 921655 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42817 42817
[junit4] 2> 921712 INFO (Thread-1324) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42817 42817
[junit4] 2> 921713 INFO (TEST-ZkStateWriterTest.testExternalModificationToSharedClusterState-seed#[51A4D395291ABC69]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testExternalModificationToSharedClusterState
[junit4] 2> 921714 INFO (SUITE-ZkStateWriterTest-seed#[51A4D395291ABC69]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> Jul 24, 2016 7:44:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 7 leaked thread(s).
[junit4] 2> Jul 24, 2016 7:44:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest:
[junit4] 2> 1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
[junit4] 2> at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
[junit4] 2> at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
[junit4] 2> at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
[junit4] 2> Jul 24, 2016 7:44:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
[junit4] 2> Jul 24, 2016 7:44:23 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
[junit4] 2> at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
[junit4] 2> at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
[junit4] 2> at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
[junit4] 2> NOTE: test params are: codec=Lucene62, sim=RandomSimilarity(queryNorm=true): {}, locale=ewo-CM, timezone=America/Kentucky/Monticello
[junit4] 2> NOTE: Linux 4.4.0-31-generic i386/Oracle Corporation 9-ea (32-bit)/cpus=12,threads=2,free=98739592,total=358223872
[junit4] 2> NOTE: All tests run in this JVM: [TestDynamicFieldResource, DocValuesTest, BigEndianAscendingWordSerializerTest, NumericFieldsTest, TestBinaryResponseWriter, SolrCLIZkUtilsTest, CollectionsAPIDistributedZkTest, MoreLikeThisHandlerTest, LoggingHandlerTest, DistanceUnitsTest, CollectionsAPIAsyncDistributedZkTest, DeleteReplicaTest, HdfsNNFailoverTest, HdfsChaosMonkeyNothingIsSafeTest, TestBM25SimilarityFactory, ClassificationUpdateProcessorFactoryTest, LeaderInitiatedRecoveryOnShardRestartTest, TestHdfsBackupRestoreCore, TestQueryTypes, TestCloudPivotFacet, TestBackupRepositoryFactory, ZkControllerTest, TestLFUCache, HdfsDirectoryTest, TestQuerySenderNoQuery, SSLMigrationTest, SpatialFilterTest, SpellCheckCollatorTest, TestSchemaResource, AnalyticsQueryTest, TestXmlQParser, DistributedFacetPivotLargeTest, TestFieldCache, TestCoreContainer, TestLeaderInitiatedRecoveryThread, SolrCmdDistributorTest, TestSolrDeletionPolicy2, DistributedTermsComponentTest, DistributedQueueTest, TestMaxScoreQueryParser, AsyncMigrateRouteKeyTest, CachingDirectoryFactoryTest, FastVectorHighlighterTest, SpatialRPTFieldTypeTest, IndexBasedSpellCheckerTest, MBeansHandlerTest, TestReloadDeadlock, TestTrackingShardHandlerFactory, TestUpdate, HdfsTlogReplayBufferedWhileIndexingTest, TimeZoneUtilsTest, TestGraphTermsQParserPlugin, BasicFunctionalityTest, TestDefaultStatsCache, TestFiltering, ExitableDirectoryReaderTest, BJQParserTest, DirectSolrSpellCheckerTest, TestCopyFieldCollectionResource, TestCryptoKeys, TestSurroundQueryParser, TestStressVersions, TestNumericTerms64, TestFastLRUCache, TestStressCloudBlindAtomicUpdates, MultiTermTest, NotRequiredUniqueKeyTest, ZkCLITest, HighlighterTest, AliasIntegrationTest, ReplicationFactorTest, TestRawTransformer, DistribCursorPagingTest, SimplePostToolTest, DistributedFacetPivotSmallTest, ClusterStateUpdateTest, ConvertedLegacyTest, HttpPartitionTest, ZkStateWriterTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ZkStateWriterTest -Dtests.seed=51A4D395291ABC69 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ewo-CM -Dtests.timezone=America/Kentucky/Monticello -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | ZkStateWriterTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.overseer.ZkStateWriterTest:
[junit4] > 1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
[junit4] > at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
[junit4] > at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
[junit4] > at __randomizedtesting.SeedInfo.seed([51A4D395291ABC69]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=6560, name=watches-1061-thread-1, state=TIMED_WAITING, group=TGRP-ZkStateWriterTest]
[junit4] > at jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:230)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:461)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1082)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1143)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:632)
[junit4] > at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
[junit4] > at __randomizedtesting.SeedInfo.seed([51A4D395291ABC69]:0)
[junit4] Completed [274/623 (1!)] on J1 in 23.96s, 5 tests, 2 errors <<< FAILURES!
[...truncated 50726 lines...]
[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_92) - Build #
17355 - Failure!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17355/
Java: 64bit/jdk1.8.0_92 -XX:+UseCompressedOops -XX:+UseG1GC
5 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.TestLBHttpSolrClient
Error Message:
42 threads leaked from SUITE scope at org.apache.solr.client.solrj.TestLBHttpSolrClient: 1) Thread[id=1007, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=1014, name=Scheduler-96571470, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=1032, name=qtp215482348-1032, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=998, name=qtp585054900-998, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=1044, name=Thread-199, state=WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291) 6) Thread[id=1039, name=qtp215482348-1039, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 7) Thread[id=1003, name=qtp585054900-1003, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=1010, name=Thread-181, state=WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291) 9) Thread[id=1052, name=qtp612269546-1052, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 10) Thread[id=997, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 11) Thread[id=1084, name=qtp215482348-1084, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 12) Thread[id=1049, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 13) Thread[id=1001, name=qtp585054900-1001, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 14) Thread[id=1086, name=qtp215482348-1086, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 15) Thread[id=1002, name=qtp585054900-1002, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 16) Thread[id=1005, name=qtp585054900-1005-acceptor-0@644a6941-ServerConnector@b8f068f{SSL,[ssl, http/1.1]}{127.0.0.1:36225}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 17) Thread[id=1048, name=Scheduler-583147894, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 18) Thread[id=999, name=qtp585054900-999, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 19) Thread[id=1013, name=searcherExecutor-412-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 20) Thread[id=1004, name=qtp585054900-1004, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 21) Thread[id=1035, name=qtp215482348-1035, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 22) Thread[id=1064, name=Thread-208, state=WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291) 23) Thread[id=1053, name=qtp612269546-1053, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 24) Thread[id=1061, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 25) Thread[id=1056, name=qtp612269546-1056-acceptor-0@750a3b04-ServerConnector@30339c50{SSL,[ssl, http/1.1]}{127.0.0.1:34595}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 26) Thread[id=1059, name=qtp612269546-1059, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 27) Thread[id=1036, name=qtp215482348-1036-acceptor-0@3c5d77b8-ServerConnector@3a67820e{SSL,[ssl, http/1.1]}{127.0.0.1:42267}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 28) Thread[id=1037, name=qtp215482348-1037, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 29) Thread[id=1041, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 30) Thread[id=1057, name=qtp612269546-1057, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 31) Thread[id=1062, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 32) Thread[id=1000, name=qtp585054900-1000, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 33) Thread[id=1034, name=qtp215482348-1034, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 34) Thread[id=1047, name=searcherExecutor-432-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 35) Thread[id=1055, name=qtp612269546-1055, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 36) Thread[id=1008, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 37) Thread[id=1067, name=searcherExecutor-444-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 38) Thread[id=1042, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 39) Thread[id=1054, name=qtp612269546-1054, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) 40) Thread[id=1058, name=qtp612269546-1058, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) 41) Thread[id=1068, name=Scheduler-1411297016, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 42) Thread[id=1085, name=qtp215482348-1085, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 42 threads leaked from SUITE scope at org.apache.solr.client.solrj.TestLBHttpSolrClient:
1) Thread[id=1007, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=1014, name=Scheduler-96571470, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=1032, name=qtp215482348-1032, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=998, name=qtp585054900-998, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=1044, name=Thread-199, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
6) Thread[id=1039, name=qtp215482348-1039, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
7) Thread[id=1003, name=qtp585054900-1003, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
8) Thread[id=1010, name=Thread-181, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
9) Thread[id=1052, name=qtp612269546-1052, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
10) Thread[id=997, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
11) Thread[id=1084, name=qtp215482348-1084, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
12) Thread[id=1049, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
13) Thread[id=1001, name=qtp585054900-1001, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
14) Thread[id=1086, name=qtp215482348-1086, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
15) Thread[id=1002, name=qtp585054900-1002, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
16) Thread[id=1005, name=qtp585054900-1005-acceptor-0@644a6941-ServerConnector@b8f068f{SSL,[ssl, http/1.1]}{127.0.0.1:36225}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
17) Thread[id=1048, name=Scheduler-583147894, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
18) Thread[id=999, name=qtp585054900-999, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
19) Thread[id=1013, name=searcherExecutor-412-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20) Thread[id=1004, name=qtp585054900-1004, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
21) Thread[id=1035, name=qtp215482348-1035, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
22) Thread[id=1064, name=Thread-208, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
23) Thread[id=1053, name=qtp612269546-1053, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
24) Thread[id=1061, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
25) Thread[id=1056, name=qtp612269546-1056-acceptor-0@750a3b04-ServerConnector@30339c50{SSL,[ssl, http/1.1]}{127.0.0.1:34595}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
26) Thread[id=1059, name=qtp612269546-1059, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
27) Thread[id=1036, name=qtp215482348-1036-acceptor-0@3c5d77b8-ServerConnector@3a67820e{SSL,[ssl, http/1.1]}{127.0.0.1:42267}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:382)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
28) Thread[id=1037, name=qtp215482348-1037, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
29) Thread[id=1041, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
30) Thread[id=1057, name=qtp612269546-1057, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
31) Thread[id=1062, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
32) Thread[id=1000, name=qtp585054900-1000, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
33) Thread[id=1034, name=qtp215482348-1034, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
34) Thread[id=1047, name=searcherExecutor-432-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
35) Thread[id=1055, name=qtp612269546-1055, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
36) Thread[id=1008, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
37) Thread[id=1067, name=searcherExecutor-444-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
38) Thread[id=1042, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:745)
39) Thread[id=1054, name=qtp612269546-1054, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
40) Thread[id=1058, name=qtp612269546-1058, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
at java.lang.Thread.run(Thread.java:745)
41) Thread[id=1068, name=Scheduler-1411297016, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
42) Thread[id=1085, name=qtp215482348-1085, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([CFD34ACCA70DBFB9]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.TestLBHttpSolrClient
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=1064, name=Thread-208, state=WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291) 2) Thread[id=1067, name=searcherExecutor-444-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=1014, name=Scheduler-96571470, state=WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=1044, name=Thread-199, state=WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291) 5) Thread[id=1068, name=Scheduler-1411297016, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 6) Thread[id=1048, name=Scheduler-583147894, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 7) Thread[id=1047, name=searcherExecutor-432-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=1010, name=Thread-181, state=WAITING, group=TGRP-TestLBHttpSolrClient] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291) 9) Thread[id=1013, name=searcherExecutor-412-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=1064, name=Thread-208, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
2) Thread[id=1067, name=searcherExecutor-444-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=1014, name=Scheduler-96571470, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=1044, name=Thread-199, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
5) Thread[id=1068, name=Scheduler-1411297016, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
6) Thread[id=1048, name=Scheduler-583147894, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
7) Thread[id=1047, name=searcherExecutor-432-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
8) Thread[id=1010, name=Thread-181, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
9) Thread[id=1013, name=searcherExecutor-412-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([CFD34ACCA70DBFB9]:0)
FAILED: org.apache.solr.client.solrj.TestLBHttpSolrClient.testReliability
Error Message:
Java heap space
Stack Trace:
java.lang.OutOfMemoryError: Java heap space
FAILED: org.apache.solr.client.solrj.TestLBHttpSolrClient.testTwoServers
Error Message:
Captured an uncaught exception in thread: Thread[id=1051, name=aliveCheckExecutor-169-thread-1, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1051, name=aliveCheckExecutor-169-thread-1, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader
Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:37276/flmq/j/forceleader_test_collection_shard1_replica2]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:37276/flmq/j/forceleader_test_collection_shard1_replica2]
at __randomizedtesting.SeedInfo.seed([1CBA86B313CA9B82:FA2DB2732A4862E3]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:753)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1151)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1040)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:976)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
at org.apache.solr.cloud.ForceLeaderTest.sendDoc(ForceLeaderTest.java:424)
at org.apache.solr.cloud.ForceLeaderTest.assertSendDocFails(ForceLeaderTest.java:315)
at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:110)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:37276/flmq/j/forceleader_test_collection_shard1_replica2]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:393)
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:747)
... 49 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:37276/flmq/j/forceleader_test_collection_shard1_replica2
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:613)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:382)
... 50 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:37276 [/127.0.0.1] failed: Connection refused
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
... 54 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
... 64 more
Build Log:
[...truncated 11965 lines...]
[junit4] Suite: org.apache.solr.cloud.ForceLeaderTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ForceLeaderTest_1CBA86B313CA9B82-001/init-core-data-001
[junit4] 2> 1318499 INFO (SUITE-ForceLeaderTest-seed#[1CBA86B313CA9B82]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 1318499 INFO (SUITE-ForceLeaderTest-seed#[1CBA86B313CA9B82]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /flmq/j
[junit4] 2> 1318501 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1318501 INFO (Thread-3289) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1318501 INFO (Thread-3289) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1318601 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.ZkTestServer start zk server on port:46117
[junit4] 2> 1318602 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1318602 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1318603 INFO (zkCallback-2039-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@a246c5d name:ZooKeeperConnection Watcher:127.0.0.1:46117 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1318603 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1318603 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1318603 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.SolrZkClient makePath: /solr
[junit4] 2> 1318604 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
[junit4] 2> 1318605 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
[junit4] 2> 1318606 INFO (zkCallback-2040-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@34a8c890 name:ZooKeeperConnection Watcher:127.0.0.1:46117/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1318606 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
[junit4] 2> 1318606 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
[junit4] 2> 1318606 INFO (TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[1CBA86B313CA9B82]) [ ] o.a.s.c.c.SolrZkClient makePath:
[...truncated too long message...]
EPollSelectorImpl.java:93)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([CFD34ACCA70DBFB9]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=1064, name=Thread-208, state=WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:502)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
[junit4] > 2) Thread[id=1067, name=searcherExecutor-444-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 3) Thread[id=1014, name=Scheduler-96571470, state=WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 4) Thread[id=1044, name=Thread-199, state=WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:502)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
[junit4] > 5) Thread[id=1068, name=Scheduler-1411297016, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 6) Thread[id=1048, name=Scheduler-583147894, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 7) Thread[id=1047, name=searcherExecutor-432-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 8) Thread[id=1010, name=Thread-181, state=WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:502)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:1291)
[junit4] > 9) Thread[id=1013, name=searcherExecutor-412-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([CFD34ACCA70DBFB9]:0)
[junit4] Completed [83/83 (1!)] on J2 in 90.88s, 3 tests, 4 errors <<< FAILURES!
[...truncated 1 lines...]
[junit4] JVM J2: stdout was not empty, see: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-solrj/test/temp/junit4-J2-20160724_221859_550.sysout
[junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
[junit4] java.lang.OutOfMemoryError: Java heap space
[junit4] Dumping heap to /home/jenkins/workspace/Lucene-Solr-master-Linux/heapdumps/java_pid15214.hprof ...
[junit4] Heap dump file created [603289410 bytes in 1.437 secs]
[junit4] <<< JVM J2: EOF ----
[junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-solrj/test/temp/junit4-J2-20160724_221859_550.syserr
[junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
[junit4]
[junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "org.eclipse.jetty.server.session.HashSessionManager@243b652eTimer"
[junit4] <<< JVM J2: EOF ----
[...truncated 10379 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:763: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:715: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid15214.hprof
Total time: 61 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any