You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/06/08 19:23:16 UTC

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 635 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/635/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.search.join.BlockJoinFacetDistribTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest:     1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest: 
   1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([ADCE4A4A6F9267DD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.join.BlockJoinFacetDistribTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.interrupt0(Native Method)         at java.lang.Thread.interrupt(Thread.java:923)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.interrupt0(Native Method)
        at java.lang.Thread.interrupt(Thread.java:923)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([ADCE4A4A6F9267DD]:0)




Build Log:
[...truncated 11277 lines...]
   [junit4] Suite: org.apache.solr.search.join.BlockJoinFacetDistribTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/init-core-data-001
   [junit4]   2> 1273904 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1273904 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1273905 INFO  (Thread-6166) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1273905 INFO  (Thread-6166) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1274005 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:47603
   [junit4]   2> 1274005 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274005 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274009 INFO  (zkCallback-1409-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3fee5cb5 name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274009 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274009 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274009 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 1274015 INFO  (jetty-launcher-1408-thread-2) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1274015 INFO  (jetty-launcher-1408-thread-4) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1274015 INFO  (jetty-launcher-1408-thread-3) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1274015 INFO  (jetty-launcher-1408-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1274016 INFO  (jetty-launcher-1408-thread-6) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1274017 INFO  (jetty-launcher-1408-thread-5) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1274022 INFO  (jetty-launcher-1408-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5df644c5{/solr,null,AVAILABLE}
   [junit4]   2> 1274024 INFO  (jetty-launcher-1408-thread-5) [    ] o.e.j.s.ServerConnector Started ServerConnector@5898bfe7{HTTP/1.1,[http/1.1]}{127.0.0.1:36123}
   [junit4]   2> 1274024 INFO  (jetty-launcher-1408-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@522cd1a5{/solr,null,AVAILABLE}
   [junit4]   2> 1274024 INFO  (jetty-launcher-1408-thread-5) [    ] o.e.j.s.Server Started @1277693ms
   [junit4]   2> 1274024 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36123}
   [junit4]   2> 1274024 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1274024 INFO  (jetty-launcher-1408-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@1c34460b{HTTP/1.1,[http/1.1]}{127.0.0.1:33059}
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node6'
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-1) [    ] o.e.j.s.Server Started @1277694ms
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33059}
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node1'
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274025 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274026 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274030 INFO  (jetty-launcher-1408-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3179441c{/solr,null,AVAILABLE}
   [junit4]   2> 1274030 INFO  (jetty-launcher-1408-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@5be6547d{HTTP/1.1,[http/1.1]}{127.0.0.1:39291}
   [junit4]   2> 1274030 INFO  (jetty-launcher-1408-thread-3) [    ] o.e.j.s.Server Started @1277699ms
   [junit4]   2> 1274030 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39291}
   [junit4]   2> 1274030 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1274030 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node4'
   [junit4]   2> 1274031 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1274031 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1274031 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274031 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274036 INFO  (zkCallback-1411-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28108cce name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274037 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274037 INFO  (zkCallback-1410-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@16816882 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274037 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274037 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274037 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274038 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1274038 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1274038 INFO  (jetty-launcher-1408-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e23e919{/solr,null,AVAILABLE}
   [junit4]   2> 1274043 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155317601220001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1274044 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155317601220002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@312a04a1{/solr,null,AVAILABLE}
   [junit4]   2> 1274045 INFO  (zkCallback-1412-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6aa28d33 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.e.j.s.ServerConnector Started ServerConnector@64ee9a0d{HTTP/1.1,[http/1.1]}{127.0.0.1:63118}
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.e.j.s.Server Started @1277714ms
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=63118}
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node5'
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1274045 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@6802c74e{HTTP/1.1,[http/1.1]}{127.0.0.1:52081}
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-2) [    ] o.e.j.s.Server Started @1277715ms
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52081}
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1274046 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node2'
   [junit4]   2> 1274047 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1274047 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1274047 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274047 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274049 INFO  (zkCallback-1413-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@e9bfb00 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274050 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274050 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274050 INFO  (jetty-launcher-1408-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4413d302{/solr,null,AVAILABLE}
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.e.j.s.ServerConnector Started ServerConnector@c014cc8{HTTP/1.1,[http/1.1]}{127.0.0.1:43593}
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.e.j.s.Server Started @1277720ms
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43593}
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node3'
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1274051 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274052 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274052 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1274052 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node6/.
   [junit4]   2> 1274052 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.CoreContainer New CoreContainer 1369588627
   [junit4]   2> 1274052 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node6]
   [junit4]   2> 1274053 INFO  (zkCallback-1414-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@57c5c8a name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274053 WARN  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node6/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node6/lib
   [junit4]   2> 1274053 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1274054 INFO  (zkCallback-1415-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4b87c5e2 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274054 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node4/.
   [junit4]   2> 1274054 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.CoreContainer New CoreContainer 410253114
   [junit4]   2> 1274054 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node4]
   [junit4]   2> 1274054 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1274054 WARN  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node4/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node4/lib
   [junit4]   2> 1274054 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1274055 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1274055 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1274055 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47603/solr
   [junit4]   2> 1274055 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1274055 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274055 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274055 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274056 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1274059 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274059 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274059 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274059 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node1/.
   [junit4]   2> 1274060 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 558847953
   [junit4]   2> 1274060 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node1]
   [junit4]   2> 1274060 WARN  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node1/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node1/lib
   [junit4]   2> 1274060 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1274061 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47603/solr
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47603/solr
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274062 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274063 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274065 INFO  (zkCallback-1424-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@68a6c1ce name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274066 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1274066 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274066 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node5/.
   [junit4]   2> 1274067 INFO  (jetty-launcher-1408-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274067 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.CoreContainer New CoreContainer 129809147
   [junit4]   2> 1274067 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node5]
   [junit4]   2> 1274067 WARN  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node5/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node5/lib
   [junit4]   2> 1274067 INFO  (zkCallback-1418-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c1a2f82 name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274067 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1274068 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1274068 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1274069 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1274069 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47603/solr
   [junit4]   2> 1274069 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1274069 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274069 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274073 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274073 INFO  (jetty-launcher-1408-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274075 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$37/1637105727@1e801258
   [junit4]   2> 1274075 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node3/.
   [junit4]   2> 1274075 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.CoreContainer New CoreContainer 1279548196
   [junit4]   2> 1274075 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274075 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node3]
   [junit4]   2> 1274076 WARN  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node3/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node3/lib
   [junit4]   2> 1274076 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1274077 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1274077 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1274077 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1274077 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47603/solr
   [junit4]   2> 1274077 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1274078 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274078 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274078 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$37/1637105727@6d052b2f
   [junit4]   2> 1274079 INFO  (zkCallback-1427-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5441cff9 name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274080 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274081 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274081 INFO  (jetty-launcher-1408-thread-6) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274083 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@d7a5226 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274083 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274084 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2c692424 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274085 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274086 INFO  (zkCallback-1431-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@49ef754b name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274086 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274087 INFO  (jetty-launcher-1408-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274088 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$37/1637105727@4f658c9b
   [junit4]   2> 1274088 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274090 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@710add17 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274090 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274091 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1274092 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1274092 INFO  (zkCallback-1423-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@764b5554 name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274093 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274094 INFO  (jetty-launcher-1408-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274094 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1274095 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$37/1637105727@23f92078
   [junit4]   2> 1274095 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274098 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15531760122000a, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1274099 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@27259143 name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274099 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274101 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155317601220008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1274101 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155317601220005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1274103 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1274104 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1274104 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node2/.
   [junit4]   2> 1274104 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.CoreContainer New CoreContainer 1942372304
   [junit4]   2> 1274104 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node2]
   [junit4]   2> 1274104 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$37/1637105727@23977881
   [junit4]   2> 1274104 WARN  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node2/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node2/lib
   [junit4]   2> 1274105 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1274105 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274106 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1274107 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1274107 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1274107 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47603/solr
   [junit4]   2> 1274107 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1274107 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1274107 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274108 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1274110 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1274112 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1274114 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1274115 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1274116 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1274117 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1274118 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1274119 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1274120 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1274122 INFO  (zkCallback-1438-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@65ec17b9 name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274122 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274122 INFO  (jetty-launcher-1408-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1274122 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@14b0ce5d name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274125 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$37/1637105727@929071f
   [junit4]   2> 1274125 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1274126 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274127 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1274130 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1274131 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@30396d6d name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1274131 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1274131 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1274134 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1274135 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96037349953241102-127.0.0.1:43593_solr-n_0000000000
   [junit4]   2> 1274135 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43593_solr
   [junit4]   2> 1274135 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1274136 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1274137 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1274138 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer Overseer (id=96037349953241102-127.0.0.1:43593_solr-n_0000000000) starting
   [junit4]   2> 1274138 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1274138 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1274140 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1274141 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1274142 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1274143 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1274144 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1274145 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1274145 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1274145 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1274146 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96037349953241103-127.0.0.1:63118_solr-n_0000000001
   [junit4]   2> 1274146 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1274146 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1274147 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1274147 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96037349953241102-127.0.0.1:43593_solr-n_0000000000 to know if I could be the leader
   [junit4]   2> 1274148 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1274148 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63118_solr
   [junit4]   2> 1274149 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:63118_solr
   [junit4]   2> 1274151 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1274152 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1274152 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1274152 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1274151 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1274154 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1274154 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1274155 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1274155 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1274155 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96037349953241100-127.0.0.1:36123_solr-n_0000000002
   [junit4]   2> 1274156 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1274157 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1274158 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1274155 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1274160 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1274160 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96037349953241101-127.0.0.1:33059_solr-n_0000000003
   [junit4]   2> 1274160 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1274162 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96037349953241106-127.0.0.1:52081_solr-n_0000000004
   [junit4]   2> 1274162 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96037349953241103-127.0.0.1:63118_solr-n_0000000001 to know if I could be the leader
   [junit4]   2> 1274162 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96037349953241100-127.0.0.1:36123_solr-n_0000000002 to know if I could be the leader
   [junit4]   2> 1274163 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:33059_solr as DOWN
   [junit4]   2> 1274164 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:36123_solr as DOWN
   [junit4]   2> 1274164 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96037349953241101-127.0.0.1:33059_solr-n_0000000003 to know if I could be the leader
   [junit4]   2> 1274165 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1274166 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:52081_solr as DOWN
   [junit4]   2> 1274167 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1274168 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96037349953241104-127.0.0.1:39291_solr-n_0000000005
   [junit4]   2> 1274169 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1274169 INFO  (OverseerCollectionConfigSetProcessor-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1274170 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1274170 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96037349953241106-127.0.0.1:52081_solr-n_0000000004 to know if I could be the leader
   [junit4]   2> 1274170 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:43593_solr as DOWN
   [junit4]   2> 1274171 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:39291_solr as DOWN
   [junit4]   2> 1274176 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 1274177 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer processMessage: queueSize: 5, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:33059_solr"} current state version: 0
   [junit4]   2> 1274177 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:33059_solr
   [junit4]   2> 1274180 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:36123_solr"} current state version: 0
   [junit4]   2> 1274180 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:36123_solr
   [junit4]   2> 1274181 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:52081_solr"} current state version: 0
   [junit4]   2> 1274181 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:52081_solr
   [junit4]   2> 1274182 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:39291_solr"} current state version: 0
   [junit4]   2> 1274182 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:39291_solr
   [junit4]   2> 1274183 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.Overseer processMessage: queueSize: 0, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:43593_solr"} current state version: 0
   [junit4]   2> 1274183 INFO  (OverseerStateUpdate-96037349953241102-127.0.0.1:43593_solr-n_0000000000) [n:127.0.0.1:43593_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:43593_solr
   [junit4]   2> 1274185 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1274185 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1274187 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node5/.
   [junit4]   2> 1274187 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1274188 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1274188 INFO  (jetty-launcher-1408-thread-6) [n:127.0.0.1:63118_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1275168 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33059_solr
   [junit4]   2> 1275168 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36123_solr
   [junit4]   2> 1275169 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:33059_solr
   [junit4]   2> 1275169 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:36123_solr
   [junit4]   2> 1275170 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52081_solr
   [junit4]   2> 1275171 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1275171 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1275171 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1275171 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1275171 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:52081_solr
   [junit4]   2> 1275171 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1275172 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1275172 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1275172 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1275172 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1275173 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1275173 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1275173 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 1275174 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1275174 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1275174 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1275175 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1275175 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1275175 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1275175 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1275176 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1275176 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1275176 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1275178 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1275179 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43593_solr
   [junit4]   2> 1275180 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39291_solr
   [junit4]   2> 1275182 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:39291_solr
   [junit4]   2> 1275182 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1275182 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:43593_solr
   [junit4]   2> 1275191 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1275192 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1275192 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1275192 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1275192 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1275193 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1275193 INFO  (zkCallback-1440-thread-1-processing-n:127.0.0.1:52081_solr) [n:127.0.0.1:52081_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 1275194 INFO  (zkCallback-1428-thread-1-processing-n:127.0.0.1:33059_solr) [n:127.0.0.1:33059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 1275198 INFO  (zkCallback-1433-thread-1-processing-n:127.0.0.1:43593_solr) [n:127.0.0.1:43593_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 1275198 INFO  (zkCallback-1435-thread-1-processing-n:127.0.0.1:39291_solr) [n:127.0.0.1:39291_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 1275198 INFO  (zkCallback-1432-thread-1-processing-n:127.0.0.1:36123_solr) [n:127.0.0.1:36123_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 1275199 INFO  (zkCallback-1434-thread-1-processing-n:127.0.0.1:63118_solr) [n:127.0.0.1:63118_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (6)
   [junit4]   2> 1275204 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1275205 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1275205 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node6/.
   [junit4]   2> 1275206 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1275210 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1275210 INFO  (jetty-launcher-1408-thread-5) [n:127.0.0.1:36123_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1275219 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1275219 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1275220 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node4/.
   [junit4]   2> 1275220 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1275221 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1275221 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1275222 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node1/.
   [junit4]   2> 1275222 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1275223 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1275223 INFO  (jetty-launcher-1408-thread-3) [n:127.0.0.1:39291_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1275223 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1275223 INFO  (jetty-launcher-1408-thread-1) [n:127.0.0.1:33059_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1275240 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1275240 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1275241 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node2/.
   [junit4]   2> 1275242 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1275243 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1275243 INFO  (jetty-launcher-1408-thread-2) [n:127.0.0.1:52081_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1275254 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1275254 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1275254 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.search.join.BlockJoinFacetDistribTest_ADCE4A4A6F9267DD-001/tempDir-001/node3/.
   [junit4]   2> 1275254 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1275255 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 1275255 INFO  (jetty-launcher-1408-thread-4) [n:127.0.0.1:43593_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1275256 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1275257 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1275258 INFO  (zkCallback-1446-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@290fae5e name:ZooKeeperConnection Watcher:127.0.0.1:47603 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1275259 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1275259 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1275261 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:47603/solr
   [junit4]   2> 1275262 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1275262 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1275263 INFO  (zkCallback-1449-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@204fe04a name:ZooKeeperConnection Watcher:127.0.0.1:47603/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1275263 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1275263 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1275263 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1275264 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 1275266 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (6)
   [junit4]   2> 1275268 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-copyfield-test.xml
   [junit4]   2> 1275272 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-currency-ft-oer-norates.xml
   [junit4]   2> 1275274 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-slave.xml
   [junit4]   2> 1275277 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-cdcr.xml
   [junit4]   2> 1275280 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-HighlighterMaxOffsetTest.xml
   [junit4]   2> 1275282 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/phrasesuggest.txt
   [junit4]   2> 1275285 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-classification.xml
   [junit4]   2> 1275287 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-script-updateprocessor.xml
   [junit4]   2> 1275290 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-init-error.xml
   [junit4]   2> 1275292 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-class-name-shortening-on-serialization.xml
   [junit4]   2> 1275295 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-cdcrupdatelog.xml
   [junit4]   2> 1275298 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-logmergepolicy.xml
   [junit4]   2> 1275300 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-dup-field.xml
   [junit4]   2> 1275303 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-currency-dynamic-multivalued.xml
   [junit4]   2> 1275305 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-docValuesMulti.xml
   [junit4]   2> 1275308 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-lmjelinekmercer.xml
   [junit4]   2> 1275310 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-tieredmergepolicyfactory.xml
   [junit4]   2> 1275313 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-sweetspot-partial-norms.xml
   [junit4]   2> 1275315 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-collapseqparser.xml
   [junit4]   2> 1275318 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-externalversionconstraint.xml
   [junit4]   2> 1275321 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-dynamicfield-required.xml
   [junit4]   2> 1275323 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-complex-phrase.xml
   [junit4]   2> 1275326 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/protwords.txt
   [junit4]   2> 1275328 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/old_synonyms.txt
   [junit4]   2> 1275331 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-docValuesMissing.xml
   [junit4]   2> 1275333 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-bogus-analysis-parameters.xml
   [junit4]   2> 1275336 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-eff.xml
   [junit4]   2> 1275339 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-rest.xml
   [junit4]   2> 1275341 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-distrib-interval-faceting.xml
   [junit4]   2> 1275344 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-uniquekey-multivalued.xml
   [junit4]   2> 1275347 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/stop-1.txt
   [junit4]   2> 1275349 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-sorts.xml
   [junit4]   2> 1275352 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-querysender.xml
   [junit4]   2> 1275354 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-sim-global-vs-ft-mismatch.xml
   [junit4]   2> 1275357 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-schema-currency-multivalued.xml
   [junit4]   2> 1275360 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-minimal.xml
   [junit4]   2> 1275363 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/solrconfig-plugcollector.xml
   [junit4]   2> 1275365 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/bad-mpf-solrconfig.xml
   [junit4]   2> 1275367 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/da_UTF8.xml
   [junit4]   2> 1275370 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/solrCloudCollectionConfig/schema-replication2.xml
   [junit4]   2> 1275373

[...truncated too long message...]

(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1289216 INFO  (jetty-launcher-1408-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@522cd1a5{/solr,null,UNAVAILABLE}
   [junit4]   2> 1289223 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47603 47603
   [junit4]   2> 1289276 INFO  (Thread-6166) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47603 47603
   [junit4]   2> 1289278 WARN  (Thread-6166) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/aliases.json
   [junit4]   2> 	7	/solr/clusterprops.json
   [junit4]   2> 	6	/solr/security.json
   [junit4]   2> 	6	/solr/configs/solrCloudCollectionConfig
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/clusterstate.json
   [junit4]   2> 	6	/solr/collections/facetcollection/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/live_nodes
   [junit4]   2> 	7	/solr/collections
   [junit4]   2> 
   [junit4]   2> 1289278 INFO  (SUITE-BlockJoinFacetDistribTest-seed#[ADCE4A4A6F9267DD]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Jun 08, 2016 7:21:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Jun 08, 2016 7:21:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest: 
   [junit4]   2>    1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Jun 08, 2016 7:21:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2> Jun 08, 2016 7:21:43 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2>         at java.lang.Thread.interrupt0(Native Method)
   [junit4]   2>         at java.lang.Thread.interrupt(Thread.java:923)
   [junit4]   2>         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {type_s=PostingsFormat(name=Memory doPackFST= true), COLOR_s=Lucene50(blocksize=128), SIZE_s=PostingsFormat(name=Memory doPackFST= false), _version_=PostingsFormat(name=Memory doPackFST= true), _root_=PostingsFormat(name=Memory doPackFST= true), BRAND_s=Lucene50(blocksize=128), id=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1163, maxMBSortInHeap=6.995252896809159, sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=no-NO, timezone=Asia/Hebron
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=2,free=69309520,total=398458880
   [junit4]   2> NOTE: All tests run in this JVM: [TestIBSimilarityFactory, HighlighterMaxOffsetTest, BasicDistributedZkTest, SuggestComponentTest, TestLMDirichletSimilarityFactory, TestCollapseQParserPlugin, TestSubQueryTransformer, HdfsUnloadDistributedZkTest, TestReplicaProperties, TestScoreJoinQPNoScore, HdfsBasicDistributedZk2Test, CircularListTest, TestManagedResourceStorage, AlternateDirectoryTest, HLLSerializationTest, TestLRUStatsCache, TestMinMaxOnMultiValuedField, TestSubQueryTransformerDistrib, TestGroupingSearch, SchemaVersionSpecificBehaviorTest, HdfsDirectoryFactoryTest, TestRecoveryHdfs, CursorPagingTest, CloudExitableDirectoryReaderTest, BufferStoreTest, TestQueryWrapperFilter, SliceStateTest, DirectoryFactoryTest, TestPostingsSolrHighlighter, RequestLoggingTest, TestRandomRequestDistribution, PreAnalyzedUpdateProcessorTest, TestFieldTypeCollectionResource, ReplicationFactorTest, TestLockTree, TestSha256AuthenticationProvider, UUIDFieldTest, ClusterStateTest, TestAnalyzedSuggestions, TestXmlQParser, SolrIndexConfigTest, TestEmbeddedSolrServerConstructors, IndexBasedSpellCheckerTest, InfoHandlerTest, TestNRTOpen, TestCloudInspectUtil, ConcurrentDeleteAndCreateCollectionTest, TestTolerantUpdateProcessorRandomCloud, VersionInfoTest, TestSolrConfigHandlerConcurrent, ClusterStateUpdateTest, TimeZoneUtilsTest, TestConfig, TestShortCircuitedRequests, TestFilteredDocIdSet, TestRequestStatusCollectionAPI, TestHashPartitioner, TestUninvertingReader, PeerSyncTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SpatialFilterTest, TestDefaultStatsCache, DocValuesMultiTest, RulesTest, LeaderInitiatedRecoveryOnCommitTest, TestDownShardTolerantSearch, TestInfoStreamLogging, TestCursorMarkWithoutUniqueKey, TestWordDelimiterFilterFactory, TolerantUpdateProcessorTest, DataDrivenBlockJoinTest, PKIAuthenticationIntegrationTest, AnalyticsMergeStrategyTest, BadCopyFieldTest, DistribJoinFromCollectionTest, ShardRoutingTest, ZkStateReaderTest, SpatialHeatmapFacetsTest, TestFreeTextSuggestions, TestPKIAuthenticationPlugin, TestUpdate, TestFieldCacheSortRandom, TestCloudBackupRestore, HdfsRestartWhileUpdatingTest, TestSolrCloudWithKerberosAlt, XsltUpdateRequestHandlerTest, TestComponentsName, DistributedSpellCheckComponentTest, SoftAutoCommitTest, DOMUtilTest, RecoveryAfterSoftCommitTest, RollingRestartTest, TestSolrDeletionPolicy1, SolrXmlInZkTest, SaslZkACLProviderTest, TriLevelCompositeIdRoutingTest, TestDocSet, SuggesterFSTTest, TestJmxIntegration, TestSerializedLuceneMatchVersion, TestJsonRequest, DistributedFacetPivotSmallTest, TestStressVersions, HttpPartitionTest, TestFieldCollectionResource, TestMaxScoreQueryParser, TestConfigSetProperties, TestExactSharedStatsCache, BlockJoinFacetDistribTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BlockJoinFacetDistribTest -Dtests.seed=ADCE4A4A6F9267DD -Dtests.slow=true -Dtests.locale=no-NO -Dtests.timezone=Asia/Hebron -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | BlockJoinFacetDistribTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest: 
   [junit4]    >    1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ADCE4A4A6F9267DD]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=13205, name=OverseerHdfsCoreFailoverThread-96037349953241106-127.0.0.1:52081_solr-n_0000000004, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.interrupt0(Native Method)
   [junit4]    >         at java.lang.Thread.interrupt(Thread.java:923)
   [junit4]    >         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ADCE4A4A6F9267DD]:0)
   [junit4] Completed [261/612 (1!)] on J0 in 38.41s, 1 test, 2 errors <<< FAILURES!

[...truncated 1111 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:531: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1427: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2560 tests, 2 suite-level errors, 1514 ignored (61 assumptions) [seed: ADCE4A4A6F9267DD]

Total time: 43 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any