You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/10/04 12:01:38 UTC

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1453 - Still Failing!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1453/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry

Error Message:
Captured an uncaught exception in thread: Thread[id=25503, name=OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25503, name=OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]
	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893:E761CAB8C9D6AB0B]:0)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893]:0)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
	... 5 more




Build Log:
[...truncated 13277 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestLeaderElectionZkExpiry
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestLeaderElectionZkExpiry_31BEEBE2FFCBB893-001/init-core-data-001
   [junit4]   2> 2803362 WARN  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 2803363 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2803364 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2803364 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2803370 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLeaderElectionWithZkExpiry
   [junit4]   2> 2803373 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2803419 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac
   [junit4]   2> 2803426 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac
   [junit4]   2> 2803426 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac
   [junit4]   2> 2803427 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestLeaderElectionZkExpiry_31BEEBE2FFCBB893-001/testLeaderElectionWithZkExpiry-solr-001
   [junit4]   2> 2803428 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2803428 INFO  (Thread-5763) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2803428 INFO  (Thread-5763) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2803429 ERROR (Thread-5763) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2803528 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.ZkTestServer start zk server on port:64902
   [junit4]   2> 2803659 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2803660 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
   [junit4]   2> 2803661 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530626-dummy.host.com:8984_solr-n_0000000000) starting
   [junit4]   2> 2803669 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
   [junit4]   2> 2803670 INFO  (zkCallback-4442-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2803674 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
   [junit4]   2> java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:182)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2803675 WARN  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Session 0x15ee72978f20002 for server 127.0.0.1/127.0.0.1:64902, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.io.IOException: Connection reset by peer
   [junit4]   2> 	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
   [junit4]   2> 	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
   [junit4]   2> 	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
   [junit4]   2> 2803776 WARN  (zkCallback-4442-thread-2-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2803776 WARN  (zkCallback-4442-thread-2-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2805232 WARN  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20002 has expired
   [junit4]   2> 2805232 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2805232 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2805232 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530626-dummy.host.com:8984_solr-n_0000000000) closing
   [junit4]   2> 2805232 WARN  (OverseerCollectionConfigSetProcessor-98770407157530626-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerTaskProcessor Overseer cannot talk to ZK
   [junit4]   2> 2805235 INFO  (OverseerStateUpdate-98770407157530626-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
   [junit4]   2> 2805237 ERROR (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:299)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2805276 WARN  (OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr    ] o.a.s.c.a.TriggerBase Exception checking ZK path /autoscaling/triggerState
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/triggerState
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:306)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:303)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:303)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:512)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:415)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.TriggerBase.<init>(TriggerBase.java:45)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.NodeLostTrigger.<init>(NodeLostTrigger.java:72)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScaling$TriggerFactory.create(AutoScaling.java:138)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.loadTriggers(OverseerTriggerThread.java:365)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.refreshAutoScalingConf(OverseerTriggerThread.java:312)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:137)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2805277 WARN  (zkCallback-4442-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> ???. 04, 2017 4:33:30 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000,5,Overseer autoscaling triggers]
   [junit4]   2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893]:0)
   [junit4]   2> 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
   [junit4]   2> 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2805288 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2805290 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2805290 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2805291 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 2805294 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x15ee72978f20003 due to java.nio.channels.CancelledKeyException
   [junit4]   2> 2805394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2805394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20003 has expired
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2806983 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2806983 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2807082 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:898)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:475)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 2807083 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:898)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:475)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 10 more
   [junit4]   2> 
   [junit4]   2> 2807083 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2807085 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2807087 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2807088 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20004 has expired
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2808714 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2808714 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2808813 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2808813 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2808813 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2808815 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2808822 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2808822 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20005 has expired
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2809873 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2809873 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2809981 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2809981 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20006 has expired
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2811493 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2811493 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2811592 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2811592 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2811592 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2811595 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2811603 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2811604 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2813394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20007 has expired
   [junit4]   2> 2813394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2813395 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2813395 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2813397 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2813397 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2813495 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2813495 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2813496 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2813499 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2813499 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2813500 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20008 has expired
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2815185 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2815185 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2815284 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2815284 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2815285 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2815287 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2815291 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
   [junit4]   2> 2815293 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) starting
   [junit4]   2> 2815294 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.TestLeaderElectionZkExpiry Time=1294496485260407 Overseer leader is = dummy.host.com:8984_solr
   [junit4]   2> 2815299 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
   [junit4]   2> 2815299 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) closing
   [junit4]   2> 2815300 INFO  (OverseerStateUpdate-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer According to ZK I (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 2815302 INFO  (OverseerStateUpdate-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
   [junit4]   2> 2815302 INFO  (OverseerCollectionConfigSetProcessor-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 2815304 ERROR (OverseerCollectionConfigSetProcessor-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer 
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Object.wait(Object.java:502)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:61)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:178)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2815307 WARN  (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer I'm exiting, but I'm still the leader
   [junit4]   2> 2815307 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 2815307 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2815308 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ee72978f2000a, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2816808 WARN  (zkCallback-4442-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2816808 WARN  (zkCallback-4442-thread-6-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2816809 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2816810 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:347)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2816810 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2816808 ERROR (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Could not delete my leader node 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:246)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:243)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:243)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:310)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2816814 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2816814 WARN  (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Unable to rejoinElection 
   [junit4]   2> org.apache.solr.common.SolrException: Unable to rejoin election
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:1924)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:324)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect
   [junit4]   2> 	at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:772)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerElectionContext.copy(ElectionContext.java:815)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:385)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:1921)
   [junit4]   2> 	... 2 more
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:770)
   [junit4]   2> 	... 5 more
   [junit4]   2> 2816814 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=959856624
   [junit4]   2> 2816815 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2816815 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@892c757: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5b81e7bc
   [junit4]   2> 2816818 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2816818 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@42733cb6: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@49ffb006
   [junit4]   2> 2816821 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2816821 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@50c718ed: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5aa407f0
   [junit4]   2> 2816821 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2816821 ERROR (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2816822 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64902 64902
   [junit4]   2> 2821884 INFO  (Thread-5763) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64902 64902
   [junit4]   2> 2821891 WARN  (Thread-5763) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 	4	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 2821892 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.SolrTestCaseJ4 ###Ending testLeaderElectionWithZkExpiry
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLeaderElectionZkExpiry -Dtests.method=testLeaderElectionWithZkExpiry -Dtests.seed=31BEEBE2FFCBB893 -Dtests.slow=true -Dtests.locale=mk-MK -Dtests.timezone=Asia/Tashkent -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   18.5s J0 | TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25503, name=OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893:E761CAB8C9D6AB0B]:0)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893]:0)
   [junit4]    > 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
   [junit4]    > 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
   [junit4]    > 	... 5 more
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=959856624
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2821901 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2821901 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=mk-MK, timezone=Asia/Tashkent
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=92765040,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestTolerantSearch, TestBinaryField, TestShortCircuitedRequests, TestCryptoKeys, DistributedDebugComponentTest, TestSolrCoreProperties, TestFreeTextSuggestions, TestSweetSpotSimilarityFactory, TestWriterPerf, HealthCheckHandlerTest, AnalyticsQueryTest, TestRestoreCore, MoveReplicaHDFSTest, TestCoreContainer, TestSolrQueryResponse, TestBackupRepositoryFactory, BooleanFieldTest, TestScoreJoinQPScore, SpatialFilterTest, TestUtils, TestLeaderInitiatedRecoveryThread, SortByFunctionTest, TestSlowCompositeReaderWrapper, BlockJoinFacetRandomTest, SSLMigrationTest, SolrCloudExampleTest, CdcrRequestHandlerTest, OverseerTest, TestRandomDVFaceting, PeerSyncTest, TestExportWriter, ResponseHeaderTest, TestLMDirichletSimilarityFactory, HdfsDirectoryFactoryTest, ScriptEngineTest, ConnectionReuseTest, TestSolrCloudWithKerberosAlt, TestDistributedSearch, CoreAdminRequestStatusTest, ConnectionManagerTest, TestSurroundQueryParser, AutoAddReplicasIntegrationTest, DirectSolrConnectionTest, RAMDirectoryFactoryTest, TestHttpShardHandlerFactory, V2ApiIntegrationTest, ClusterStateUpdateTest, TestAtomicUpdateErrorCases, UUIDFieldTest, TestUseDocValuesAsStored2, TestXIncludeConfig, TestFuzzyAnalyzedSuggestions, TestMissingGroups, DirectoryFactoryTest, DistributedFacetPivotSmallTest, TestExtendedDismaxParser, NumericFieldsTest, PKIAuthenticationIntegrationTest, TestSchemalessBufferedUpdates, DocValuesNotIndexedTest, TestBlendedInfixSuggestions, InfoHandlerTest, TestMacroExpander, VersionInfoTest, DistanceFunctionTest, TestSkipOverseerOperations, CurrencyFieldTypeTest, ResponseLogComponentTest, HdfsRecoverLeaseTest, ResourceLoaderTest, TestMaxTokenLenTokenizer, HLLSerializationTest, DeleteLastCustomShardedReplicaTest, TestConfigSetsAPIExclusivity, TestSha256AuthenticationProvider, TestNestedDocsSort, AtomicUpdateProcessorFactoryTest, TestXmlQParserPlugin, DeleteStatusTest, LeaderElectionTest, TestTolerantUpdateProcessorCloud, TestFieldCacheSort, TestEmbeddedSolrServerAdminHandler, TestRandomFlRTGCloud, CloudMLTQParserTest, TestSolrXml, TestCodecSupport, HdfsCollectionsAPIDistributedZkTest, DistributedSpellCheckComponentTest, TestBulkSchemaConcurrent, DistribDocExpirationUpdateProcessorTest, DefaultValueUpdateProcessorTest, TestNumericRangeQuery64, SliceStateTest, TestImpersonationWithHadoopAuth, TestSolrCoreParser, NodeLostTriggerTest, TestManagedResourceStorage, TlogReplayBufferedWhileIndexingTest, SegmentsInfoRequestHandlerTest, URLClassifyProcessorTest, TestConfigReload, GraphQueryTest, TestOrdValues, HttpTriggerListenerTest, TestSQLHandlerNonCloud, SuggesterWFSTTest, TestStandardQParsers, TestTestInjection, TestExclusionRuleCollectionAccess, SolrCoreTest, TestQuerySenderNoQuery, InfixSuggestersTest, TestReloadAndDeleteDocs, TestLFUCache, TestQuerySenderListener, TestUpdate, ChaosMonkeyNothingIsSafeTest, TestTrieFacet, AlternateDirectoryTest, HighlighterMaxOffsetTest, HdfsRestartWhileUpdatingTest, DirectUpdateHandlerOptimizeTest, LargeFieldTest, AssignTest, TestFieldTypeCollectionResource, ZkControllerTest, TestSubQueryTransformerDistrib, SolrMetricsIntegrationTest, TestAuthenticationFramework, TestExactStatsCache, BinaryUpdateRequestHandlerTest, TestConfigsApi, ExplicitHLLTest, SpellCheckCollatorWithCollapseTest, AddBlockUpdateTest, DistributedQueryComponentCustomSortTest, TestInPlaceUpdatesStandalone, TestHighlightDedupGrouping, TestRequestForwarding, SubstringBytesRefFilterTest, XmlUpdateRequestHandlerTest, XsltUpdateRequestHandlerTest, AutoScalingHandlerTest, HdfsThreadLeakTest, TestLegacyNumericUtils, RecoveryAfterSoftCommitTest, TestPushWriter, SolrMetricManagerTest, TestSubQueryTransformer, TestLegacyFieldCache, TestSerializedLuceneMatchVersion, TestLegacyNumericRangeQueryBuilder, TestReplicationHandlerBackup, AutoscalingHistoryHandlerTest, TestFilteredDocIdSet, TestPhraseSuggestions, DirectUpdateHandlerTest, TestSimpleTextCodec, DocValuesMissingTest, SparseHLLTest, LeaderInitiatedRecoveryOnShardRestartTest, TestFunctionQuery, SolrIndexSplitterTest, ZkStateReaderTest, PrimitiveFieldTypeTest, CollectionStateFormat2Test, TestNoOpRegenerator, CircularListTest, TestCustomStream, TestCursorMarkWithoutUniqueKey, TestOmitPositions, TestReversedWildcardFilterFactory, TestHdfsUpdateLog, SimpleCollectionCreateDeleteTest, TestManagedSynonymGraphFilterFactory, AliasIntegrationTest, SimplePostToolTest, BasicDistributedZkTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, ZkSolrClientTest, TestZkChroot, ZkCLITest, ShardRoutingCustomTest, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, HardAutoCommitTest, SimpleFacetsTest, SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, QueryElevationComponentTest, TestSort, TestLazyCores, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, TestSolrQueryParser, RegexBoostProcessorTest, TestJmxIntegration, ReturnFieldsTest, QueryParsingTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, HighlighterConfigTest, UpdateParamsTest, CopyFieldTest, SolrIndexConfigTest, TestStressRecovery, BaseCdcrDistributedZkTest, CdcrBootstrapTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CollectionsAPIAsyncDistributedZkTest, DeleteNodeTest, DeleteReplicaTest, DistribJoinFromCollectionTest, ForceLeaderTest, MissingSegmentRecoveryTest, MultiThreadedOCPTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, PeerSyncReplicationTest, SolrCLIZkUtilsTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudPivotFacet, TestCloudPseudoReturnFields, TestCloudRecovery, TestClusterProperties, TestCollectionsAPIViaSolrCloudCluster, TestDistribDocBasedVersion, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry]
   [junit4] Completed [547/745 (1!)] on J0 in 18.57s, 1 test, 1 error <<< FAILURES!

[...truncated 43623 lines...]
-ecj-javadoc-lint-src:
    [mkdir] Created dir: /var/tmp/ecj112807485
 [ecj-lint] Compiling 1149 source files to /var/tmp/ecj112807485
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. ERROR in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java (at line 27)
 [ecj-lint] 	import java.util.concurrent.atomic.AtomicBoolean;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import java.util.concurrent.atomic.AtomicBoolean is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. ERROR in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java (at line 26)
 [ecj-lint] 	import com.google.common.base.Preconditions;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import com.google.common.base.Preconditions is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/CoreContainer.java (at line 1033)
 [ecj-lint] 	core = new SolrCore(this, dcore, coreConfig);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'core' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 4. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/HdfsDirectoryFactory.java (at line 234)
 [ecj-lint] 	dir = new BlockDirectory(path, hdfsDir, cache, null, blockCacheReadEnabled, false, cacheMerges, cacheReadOnce);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'dir' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 121)
 [ecj-lint] 	reader = cfiltfac.create(reader);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'reader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 6. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 145)
 [ecj-lint] 	return namedList;
 [ecj-lint] 	^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'listBasedTokenStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 7. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java (at line 1285)
 [ecj-lint] 	DirectoryReader reader = s==null ? null : s.get().getIndexReader();
 [ecj-lint] 	                ^^^^^^
 [ecj-lint] Resource leak: 'reader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 8. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 517)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zk, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] 9. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 743)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zkHost, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java (at line 578)
 [ecj-lint] 	tvWindowStream = new OffsetWindowTokenFilter(tvStream);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tvWindowStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/request/SimpleFacets.java (at line 925)
 [ecj-lint] 	fastForRandomSet = new HashDocSet(sset.getDocs(), 0, sset.size());
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fastForRandomSet' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java (at line 33)
 [ecj-lint] 	new SmileWriter(out, request, response).writeResponse();
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/schema/OpenExchangeRatesOrgProvider.java (at line 146)
 [ecj-lint] 	ratesJsonStream = resourceLoader.openResource(ratesFileLocation);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'ratesJsonStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java (at line 1315)
 [ecj-lint] 	UninvertingReader uninvertingReader = new UninvertingReader(new ReaderWrapper(searcher.getSlowAtomicReader(), collapseField), mapping);
 [ecj-lint] 	                  ^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'uninvertingReader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 145)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 16. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 154)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 17. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 163)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/OrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/ReverseOrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 20. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java (at line 291)
 [ecj-lint] 	throw new SolrException(ErrorCode.SERVER_ERROR, "Error occurred while loading solr.xml from zookeeper", e);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'loader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/CdcrTransactionLog.java (at line 98)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 191)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 261)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 24. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 25. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 26. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 260)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 27. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 275)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 28. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 529)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 29. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] 30. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 31. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SolrLogLayout.java (at line 149)
 [ecj-lint] 	SolrQueryRequest req = requestInfo == null ? null : requestInfo.getReq();
 [ecj-lint] 	                 ^^^
 [ecj-lint] Resource leak: 'req' is never closed
 [ecj-lint] ----------
 [ecj-lint] 31 problems (2 errors, 29 warnings)

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:689: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2081: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2120: Compile failed; see the compiler error output for details.

Total time: 93 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1456 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1456/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.SystemLogListenerTest

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:232)
	at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:190)
	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.setupCluster(SystemLogListenerTest.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
	... 30 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.SystemLogListenerTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest:     1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest: 
   1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.SystemLogListenerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
        at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
        at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)




Build Log:
[...truncated 13224 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.SystemLogListenerTest
   [junit4]   2> 4011006 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_C11663E6AE8AF65E-001/init-core-data-001
   [junit4]   2> 4011007 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4011009 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 4011010 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_C11663E6AE8AF65E-001/tempDir-001
   [junit4]   2> 4011010 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4011011 INFO  (Thread-9801) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4011011 INFO  (Thread-9801) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4011017 ERROR (Thread-9801) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4011111 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64210
   [junit4]   2> 4041817 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Oct 05, 2017 7:38:13 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> Oct 05, 2017 7:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest: 
   [junit4]   2>    1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]   2>    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]   2>    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]   2>    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Oct 05, 2017 7:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]   2> 4062115 ERROR (ProcessThread(sid:0 cport:64210):) [    ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : ProcessThread(sid:0 cport:64210):
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]   2> 4062115 WARN  (Thread-9801) [    ] o.a.s.c.ZkTestServer Server interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]   2> 4062118 ERROR (ProcessThread(sid:0 cport:64210):) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4062115 ERROR (SessionTracker) [    ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : SessionTracker
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]   2> 4062118 ERROR (SessionTracker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> Oct 05, 2017 7:38:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=ca-ES, timezone=Pacific/Guadalcanal
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=3,free=262362232,total=526385152
   [junit4]   2> NOTE: All tests run in this JVM: [TestReversedWildcardFilterFactory, CollectionStateFormat2Test, TestDocBasedVersionConstraints, TestSolrQueryResponse, TestMultiValuedNumericRangeQuery, TestFuzzyAnalyzedSuggestions, HdfsThreadLeakTest, OverseerCollectionConfigSetProcessorTest, MinimalSchemaTest, TestPullReplica, TestXmlQParserPlugin, MultiTermTest, AnalyticsQueryTest, SoftAutoCommitTest, TestIntervalFaceting, AssignTest, TestUpdate, OverseerRolesTest, LeaderInitiatedRecoveryOnShardRestartTest, UnloadDistributedZkTest, TestPseudoReturnFields, TestExclusionRuleCollectionAccess, ShufflingReplicaListTransformerTest, TestCustomStream, DistributedDebugComponentTest, SolrInfoBeanTest, ConfigureRecoveryStrategyTest, SolrMetricManagerTest, TestReplicationHandler, TestWriterPerf, TestManagedResourceStorage, CoreAdminRequestStatusTest, TestReloadAndDeleteDocs, DistribDocExpirationUpdateProcessorTest, SimplePostToolTest, SubstringBytesRefFilterTest, UUIDUpdateProcessorFallbackTest, CdcrReplicationDistributedZkTest, DeleteReplicaTest, AutoAddReplicasIntegrationTest, SolrCoreTest, TestQuerySenderNoQuery, TestPKIAuthenticationPlugin, MissingSegmentRecoveryTest, DirectUpdateHandlerTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, RulesTest, TestQuerySenderListener, TestBinaryField, TestFieldTypeCollectionResource, ConnectionReuseTest, HighlighterConfigTest, SortByFunctionTest, TestSolrCoreProperties, TestStressLiveNodes, BooleanFieldTest, QueryElevationComponentTest, OverseerTaskQueueTest, AlternateDirectoryTest, SuggestComponentContextFilterQueryTest, TestXIncludeConfig, InfoHandlerTest, AnalysisAfterCoreReloadTest, TestSolrXml, IndexSchemaTest, JSONWriterTest, TestGraphMLResponseWriter, CircularListTest, BaseCdcrDistributedZkTest, SolrJmxReporterTest, DirectUpdateHandlerOptimizeTest, TestStressVersions, DirectoryFactoryTest, TestSolrDeletionPolicy2, BlockCacheTest, DistributedQueryComponentCustomSortTest, SearchHandlerTest, TestImpersonationWithHadoopAuth, TestFieldCacheSort, TestOrdValues, TestRestoreCore, ZkControllerTest, BinaryUpdateRequestHandlerTest, TestEmbeddedSolrServerAdminHandler, TestReplicationHandlerBackup, DirectSolrConnectionTest, TestShortCircuitedRequests, DeleteLastCustomShardedReplicaTest, BigEndianAscendingWordSerializerTest, ExplicitHLLTest, TestDistributedStatsComponentCardinality, TestLockTree, ShardSplitTest, TestLeaderInitiatedRecoveryThread, SolrCloudExampleTest, FastVectorHighlighterTest, StressHdfsTest, TestSkipOverseerOperations, ReplaceNodeTest, DistributedFacetExistsSmallTest, TlogReplayBufferedWhileIndexingTest, DefaultValueUpdateProcessorTest, TestHighlightDedupGrouping, NotRequiredUniqueKeyTest, TestPHPSerializedResponseWriter, TestNamedUpdateProcessors, TestSort, TestPointFields, TestNonDefinedSimilarityFactory, CoreAdminHandlerTest, TestHdfsUpdateLog, TestBackupRepositoryFactory, TestFieldCollectionResource, TestRandomFlRTGCloud, TestIndexingPerformance, TestCoreContainer, ResponseHeaderTest, TestApiFramework, DistributedSpellCheckComponentTest, BasicDistributedZkTest, TestAddFieldRealTimeGet, LeaderFailureAfterFreshStartTest, CloudExitableDirectoryReaderTest, DistributedFacetPivotSmallTest, TestQueryWrapperFilter, LeaderElectionContextKeyTest, AddBlockUpdateTest, ConcurrentDeleteAndCreateCollectionTest, TestStressRecovery, ChaosMonkeyNothingIsSafeTest, XsltUpdateRequestHandlerTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestCloudPivotFacet, TestCloudRecovery, HdfsRecoverLeaseTest, TestPhraseSuggestions, FieldAnalysisRequestHandlerTest, TestTolerantUpdateProcessorCloud, TestRawTransformer, TestDistributedSearch, OutputWriterTest, TestExactStatsCache, RecoveryAfterSoftCommitTest, RegexBoostProcessorTest, RestartWhileUpdatingTest, TestHashQParserPlugin, DistanceFunctionTest, SSLMigrationTest, MoveReplicaHDFSTest, TestExtendedDismaxParser, TestRecoveryHdfs, TestFreeTextSuggestions, ReturnFieldsTest, NoCacheHeaderTest, SuggesterTest, TestLeaderElectionWithEmptyReplica, TestFieldCacheVsDocValues, ExecutePlanActionTest, LargeFieldTest, TestConfigReload, TestLegacyField, TestSearcherReuse, TestTestInjection, VMParamsZkACLAndCredentialsProvidersTest, RAMDirectoryFactoryTest, PrimitiveFieldTypeTest, TestFoldingMultitermQuery, HdfsRestartWhileUpdatingTest, QueryParsingTest, ClusterStateTest, TestSha256AuthenticationProvider, DisMaxRequestHandlerTest, TestDistribIDF, TestSurroundQueryParser, BasicAuthIntegrationTest, UniqFieldsUpdateProcessorFactoryTest, TestZkAclsWithHadoopAuth, ParsingFieldUpdateProcessorsTest, TestInPlaceUpdatesStandalone, TestRandomDVFaceting, ZkCLITest, TestFaceting, TestRecovery, TestRealTimeGet, TestStressReorder, TestJoin, DistributedTermsComponentTest, TestSolr4Spatial, SpellCheckComponentTest, TestGroupingSearch, PeerSyncTest, TestBadConfig, ShowFileRequestHandlerTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, DocValuesTest, DirectSolrSpellCheckerTest, TestOmitPositions, TermsComponentTest, TermVectorComponentTest, RequiredFieldsTest, SolrPluginUtilsTest, UpdateRequestProcessorFactoryTest, TestAnalyzedSuggestions, TestPartialUpdateDeduplication, TestLFUCache, TestLRUCache, TestDocumentBuilder, SliceStateTest, FileUtilsTest, CursorPagingTest, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, AliasIntegrationTest, CdcrRequestHandlerTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CleanupOldIndexTest, ConnectionManagerTest, CreateCollectionCleanupTest, CustomCollectionTest, DeleteShardTest, DistribCursorPagingTest, DocValuesNotIndexedTest, HttpPartitionTest, MoveReplicaTest, NodeMutatorTest, RemoteQueryErrorTest, ReplicationFactorTest, RollingRestartTest, SimpleCollectionCreateDeleteTest, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestClusterProperties, TestCollectionsAPIViaSolrCloudCluster, TestConfigSetsAPIExclusivity, TestCryptoKeys, TestLocalFSCloudBackupRestore, TestOnReconnectListenerSupport, TestReplicaProperties, TestRequestForwarding, TestRequestStatusCollectionAPI, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressInPlaceUpdates, TestTolerantUpdateProcessorRandomCloud, ZkFailoverTest, HttpTriggerListenerTest, NodeLostTriggerTest, SystemLogListenerTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SystemLogListenerTest -Dtests.seed=C11663E6AE8AF65E -Dtests.slow=true -Dtests.locale=ca-ES -Dtests.timezone=Pacific/Guadalcanal -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | SystemLogListenerTest (suite) <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:232)
   [junit4]    > 	at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:190)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.setupCluster(SystemLogListenerTest.java:82)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
   [junit4]    > 	... 30 moreThrowable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest: 
   [junit4]    >    1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]    >    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1252)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1326)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]    >    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]    >    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)
   [junit4] Completed [564/745 (1!)] on J0 in 54.14s, 0 tests, 3 errors <<< FAILURES!

[...truncated 47455 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:706: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:693: Source checkout is dirty (unversioned/missing files) after running tests!!! Offending files:
* lucene/licenses/morfologik-ukrainian-search-3.7.5.jar.sha1

Total time: 96 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1455 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1455/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:     1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([9A88CB42154A7069]:0)




Build Log:
[...truncated 12247 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth
   [junit4]   2> 1291131 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/init-core-data-001
   [junit4]   2> 1291131 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1291132 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1291165 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/tempDir-001
   [junit4]   2> 1291165 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1291165 INFO  (Thread-3219) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1291165 INFO  (Thread-3219) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1291165 ERROR (Thread-3219) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1291265 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:48693
   [junit4]   2> 1291293 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1291293 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1291296 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@438368ab{/solr,null,AVAILABLE}
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36bbe5e0{/solr,null,AVAILABLE}
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3f8642{HTTP/1.1,[http/1.1]}{127.0.0.1:49542}
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.Server Started @1293188ms
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49542}
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@41e4895d{HTTP/1.1,[http/1.1]}{127.0.0.1:48877}
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.Server Started @1293188ms
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48877}
   [junit4]   2> 1291298 ERROR (jetty-launcher-2838-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1291298 ERROR (jetty-launcher-2838-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-05T08:29:40.581Z
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-05T08:29:40.581Z
   [junit4]   2> 1291306 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1291322 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1291325 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48693/solr
   [junit4]   2> 1291330 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1291333 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48693/solr
   [junit4]   2> 1291482 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1291484 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1291485 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:48877_solr
   [junit4]   2> 1291485 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49542_solr
   [junit4]   2> 1291486 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.Overseer Overseer (id=98775346737840134-127.0.0.1:48877_solr-n_0000000000) starting
   [junit4]   2> 1291487 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1291487 INFO  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1291496 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48877_solr
   [junit4]   2> 1291497 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1291498 INFO  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1291548 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 1291551 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=staff, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 1291555 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 1291572 INFO  (jetty-launcher-2838-thread-1-EventThread) [n:127.0.0.1:49542_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 1291573 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 1291574 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=staff, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 1291574 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 1291578 INFO  (jetty-launcher-2838-thread-2-EventThread) [n:127.0.0.1:48877_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 1291597 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 1291614 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 1291656 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48877.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291657 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49542.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291662 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48877.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291662 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48877.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49542.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49542.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/tempDir-001/node2/.
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/tempDir-001/node1/.
   [junit4]   2> 1291702 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1291703 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48693/solr ready
   [junit4]   2> 1291856 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 1291877 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 1292009 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 1292014 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1292014 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1292014 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 1292145 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 1292160 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 1292300 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 1292302 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1292302 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1292302 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4]   2> 1292433 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 1292457 WARN  (qtp776626350-14673) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
   [junit4]   2> id: invalid user name: "bar"
   [junit4]   2> 
   [junit4]   2> 1292461 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 1292612 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 1292614 INFO  (qtp776626350-14676) [n:127.0.0.1:48877_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1292614 INFO  (qtp776626350-14676) [n:127.0.0.1:48877_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1292614 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4]   2> 1292750 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 1292754 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 1292890 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 1307909 WARN  (jetty-launcher-2838-thread-1-SendThread(127.0.0.1:48693)) [n:127.0.0.1:49542_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16290ms for sessionid 0x15eeba78ee00007
   [junit4]   2> 1307910 WARN  (jetty-launcher-2838-thread-1-SendThread(127.0.0.1:48693)) [n:127.0.0.1:49542_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16245ms for sessionid 0x15eeba78ee00004
   [junit4]   2> 1307912 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 1307913 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1307913 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1307910 WARN  (jetty-launcher-2838-thread-2-SendThread(127.0.0.1:48693)) [n:127.0.0.1:48877_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16245ms for sessionid 0x15eeba78ee00006
   [junit4]   2> 1307909 WARN  (jetty-launcher-2838-thread-2-SendThread(127.0.0.1:48693)) [n:127.0.0.1:48877_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16288ms for sessionid 0x15eeba78ee00008
   [junit4]   2> 1307913 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1307920 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1308011 WARN  (zkCallback-2856-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@739709c4 name: ZooKeeperConnection Watcher:127.0.0.1:48693/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1308012 WARN  (zkCallback-2856-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1308014 INFO  (jetty-launcher-2838-thread-1-EventThread) [n:127.0.0.1:49542_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 1308016 WARN  (zkCallback-2846-thread-3-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3fadfaa6 name: ZooKeeperConnection Watcher:127.0.0.1:48693/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1308016 WARN  (zkCallback-2846-thread-3-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1308022 WARN  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@b0fd95c name: ZooKeeperConnection Watcher:127.0.0.1:48693/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1308022 WARN  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1308030 INFO  (jetty-launcher-2838-thread-2-EventThread) [n:127.0.0.1:48877_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 1308089 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 1308091 INFO  (qtp776626350-14665) [    ] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1308092 WARN  (qtp776626350-14665) [    ] o.e.j.s.ServletHandler /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication, 
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	... 26 more
   [junit4]   2> 1308096 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4] IGNOR/A 0.00s J0 | TestImpersonationWithHadoopAuth.testForwarding
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/HADOOP-9893))
   [junit4]   2> 1308806 INFO  (jetty-closer-2839-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3f8642{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1308806 INFO  (jetty-closer-2839-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@41e4895d{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1308809 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1757477537
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1002081792
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@14768568: rootName = solr_48877, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@d58eca2
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3e7445c: rootName = solr_49542, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@752a6817
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1cb50f68: rootName = solr_48877, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2853a7df
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7a2bd70c: rootName = solr_49542, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2853a7df
   [junit4]   2> 1308824 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1308824 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@51f0bb96: rootName = solr_48877, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4ee2e926
   [junit4]   2> 1308825 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1308825 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@779e0b35: rootName = solr_49542, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4ee2e926
   [junit4]   2> 1309247 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1309247 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1309248 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98775346737840134-127.0.0.1:48877_solr-n_0000000000) closing
   [junit4]   2> 1309248 INFO  (OverseerStateUpdate-98775346737840134-127.0.0.1:48877_solr-n_0000000000) [n:127.0.0.1:48877_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:48877_solr
   [junit4]   2> 1309248 INFO  (OverseerCollectionConfigSetProcessor-98775346737840134-127.0.0.1:48877_solr-n_0000000000) [n:127.0.0.1:48877_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=98775346737840134-127.0.0.1:48877_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 1309250 WARN  (OverseerAutoScalingTriggerThread-98775346737840134-127.0.0.1:48877_solr-n_0000000000) [n:127.0.0.1:48877_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1309255 WARN  (zkCallback-2850-thread-5-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1309256 ERROR (Thread[Thread-3238,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:48877_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 1309705 INFO  (jetty-closer-2839-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@36bbe5e0{/solr,null,UNAVAILABLE}
   [junit4]   2> 1309833 INFO  (zkCallback-2846-thread-4-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49542_solr
   [junit4]   2> 1309833 INFO  (zkCallback-2846-thread-5-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1309834 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1309834 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1309836 WARN  (zkCallback-2846-thread-5-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1311338 ERROR (Thread[Thread-3236,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:49542_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 1311345 INFO  (jetty-closer-2839-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@438368ab{/solr,null,UNAVAILABLE}
   [junit4]   2> 1311348 ERROR (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1311348 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48693 48693
   [junit4]   2> 1314684 INFO  (Thread-3219) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48693 48693
   [junit4]   2> 1314690 WARN  (Thread-3219) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 	3	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 
   [junit4]   2> 1314690 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Oct 05, 2017 8:30:03 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Oct 05, 2017 8:30:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   [junit4]   2>    1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2>    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 05, 2017 8:30:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2> 1334818 ERROR (jetty-launcher-2838-thread-1-EventThread) [n:127.0.0.1:49542_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 1334818 ERROR (jetty-launcher-2838-thread-2-EventThread) [n:127.0.0.1:48877_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 05, 2017 8:30:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-LY, timezone=America/Argentina/Jujuy
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=125433136,total=388169728
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsDirectoryFactoryTest, TestCustomStream, DistributedFacetPivotLongTailTest, TestPseudoReturnFields, TestSolrCloudWithKerberosAlt, TestStressVersions, TestQuerySenderListener, MissingSegmentRecoveryTest, MultiThreadedOCPTest, DistributedQueueTest, SimpleCollectionCreateDeleteTest, DateMathParserTest, SegmentsInfoRequestHandlerTest, TestSchemalessBufferedUpdates, TestGraphMLResponseWriter, ResourceLoaderTest, HardAutoCommitTest, TestSha256AuthenticationProvider, TestHdfsUpdateLog, CloudMLTQParserTest, BlockCacheTest, TestSolrQueryResponse, TestConfigSetImmutable, URLClassifyProcessorTest, ReplaceNodeTest, ZkCLITest, PeerSyncWithIndexFingerprintCachingTest, DistributedQueryComponentCustomSortTest, TestConfigReload, TestSmileRequest, DirectUpdateHandlerTest, TestFieldTypeCollectionResource, PeerSyncTest, TestBackupRepositoryFactory, SearchHandlerTest, CoreAdminHandlerTest, AutoScalingHandlerTest, TestLMDirichletSimilarityFactory, NoCacheHeaderTest, DistributedFacetPivotSmallTest, TimeZoneUtilsTest, TestCodecSupport, TestManagedSchemaAPI, AlternateDirectoryTest, TestInitParams, HdfsLockFactoryTest, GraphQueryTest, TestCSVResponseWriter, DisMaxRequestHandlerTest, CdcrRequestHandlerTest, TestLeaderElectionWithEmptyReplica, HealthCheckHandlerTest, TestSchemaNameResource, TestBinaryField, TestStressRecovery, ExecutePlanActionTest, ChaosMonkeyNothingIsSafeTest, CdcrReplicationDistributedZkTest, TestSolrCoreProperties, InfixSuggestersTest, MultiTermTest, TestAddFieldRealTimeGet, TestConfigSets, TestFieldCacheSort, LargeFieldTest, ConcurrentDeleteAndCreateCollectionTest, SimplePostToolTest, SubstringBytesRefFilterTest, ExplicitHLLTest, OutOfBoxZkACLAndCredentialsProvidersTest, AssignTest, TestSSLRandomization, TestFoldingMultitermQuery, UUIDFieldTest, TestIndexingPerformance, TestRandomDVFaceting, TestPHPSerializedResponseWriter, DistributedSpellCheckComponentTest, SpellCheckCollatorWithCollapseTest, TestNumericRangeQuery64, TestSolrCoreParser, TestCustomSort, LeaderFailureAfterFreshStartTest, ZkControllerTest, ResponseHeaderTest, TestMultiValuedNumericRangeQuery, TestReversedWildcardFilterFactory, TestFieldCollectionResource, SuggesterWFSTTest, TestCloudSchemaless, TestXIncludeConfig, TestBulkSchemaConcurrent, TestClusterProperties, TlogReplayBufferedWhileIndexingTest, SoftAutoCommitTest, MinimalSchemaTest, OverseerTest, TestFaceting, PathHierarchyTokenizerFactoryTest, SolrTestCaseJ4Test, ScriptEngineTest, DistributedFacetExistsSmallTest, ConnectionReuseTest, SolrCloudReportersTest, BasicAuthIntegrationTest, StressHdfsTest, TestConfigsApi, TestMacroExpander, ReturnFieldsTest, SSLMigrationTest, TestSkipOverseerOperations, ClusterStateTest, TestBlendedInfixSuggestions, SolrMetricsIntegrationTest, TestNamedUpdateProcessors, DocExpirationUpdateProcessorFactoryTest, RecoveryAfterSoftCommitTest, TestStressLiveNodes, TestIntervalFaceting, TestBulkSchemaAPI, TestPushWriter, TestNestedDocsSort, TestFieldCacheVsDocValues, ClassificationUpdateProcessorFactoryTest, SolrGraphiteReporterTest, NotRequiredUniqueKeyTest, TestLegacyNumericUtils, TestSimpleTextCodec, SolrIndexSplitterTest, TestPayloadCheckQParserPlugin, PKIAuthenticationIntegrationTest, FieldAnalysisRequestHandlerTest, SyncSliceTest, SolrCoreTest, TestBinaryResponseWriter, ClassificationUpdateProcessorTest, TestCloudInspectUtil, TestSolrCoreSnapshots, TestSubQueryTransformer, TestQuerySenderNoQuery, TestFunctionQuery, TestSQLHandlerNonCloud, WordBreakSolrSpellCheckerTest, WrapperMergePolicyFactoryTest, TestImpersonationWithHadoopAuth]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestImpersonationWithHadoopAuth -Dtests.seed=9A88CB42154A7069 -Dtests.slow=true -Dtests.locale=ar-LY -Dtests.timezone=America/Argentina/Jujuy -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestImpersonationWithHadoopAuth (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   [junit4]    >    1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    >    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9A88CB42154A7069]:0)
   [junit4] Completed [263/745 (1!)] on J0 in 43.71s, 10 tests, 1 error, 1 skipped <<< FAILURES!

[...truncated 44597 lines...]
-ecj-javadoc-lint-src:
    [mkdir] Created dir: /var/tmp/ecj41246448
 [ecj-lint] Compiling 1149 source files to /var/tmp/ecj41246448
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. ERROR in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java (at line 40)
 [ecj-lint] 	import java.util.concurrent.atomic.AtomicReference;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import java.util.concurrent.atomic.AtomicReference is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/CoreContainer.java (at line 1033)
 [ecj-lint] 	core = new SolrCore(this, dcore, coreConfig);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'core' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/HdfsDirectoryFactory.java (at line 234)
 [ecj-lint] 	dir = new BlockDirectory(path, hdfsDir, cache, null, blockCacheReadEnabled, false, cacheMerges, cacheReadOnce);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'dir' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 4. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 121)
 [ecj-lint] 	reader = cfiltfac.create(reader);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'reader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 145)
 [ecj-lint] 	return namedList;
 [ecj-lint] 	^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'listBasedTokenStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 6. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java (at line 1285)
 [ecj-lint] 	DirectoryReader reader = s==null ? null : s.get().getIndexReader();
 [ecj-lint] 	                ^^^^^^
 [ecj-lint] Resource leak: 'reader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 7. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 517)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zk, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] 8. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 743)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zkHost, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 9. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java (at line 578)
 [ecj-lint] 	tvWindowStream = new OffsetWindowTokenFilter(tvStream);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tvWindowStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/request/SimpleFacets.java (at line 925)
 [ecj-lint] 	fastForRandomSet = new HashDocSet(sset.getDocs(), 0, sset.size());
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fastForRandomSet' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java (at line 33)
 [ecj-lint] 	new SmileWriter(out, request, response).writeResponse();
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/schema/OpenExchangeRatesOrgProvider.java (at line 146)
 [ecj-lint] 	ratesJsonStream = resourceLoader.openResource(ratesFileLocation);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'ratesJsonStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java (at line 1315)
 [ecj-lint] 	UninvertingReader uninvertingReader = new UninvertingReader(new ReaderWrapper(searcher.getSlowAtomicReader(), collapseField), mapping);
 [ecj-lint] 	                  ^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'uninvertingReader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 145)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 154)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 16. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 163)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 17. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/OrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/ReverseOrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java (at line 291)
 [ecj-lint] 	throw new SolrException(ErrorCode.SERVER_ERROR, "Error occurred while loading solr.xml from zookeeper", e);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'loader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 20. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/CdcrTransactionLog.java (at line 98)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 191)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 261)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 24. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 25. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 260)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 26. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 275)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 27. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 529)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 28. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] 29. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 30. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SolrLogLayout.java (at line 149)
 [ecj-lint] 	SolrQueryRequest req = requestInfo == null ? null : requestInfo.getReq();
 [ecj-lint] 	                 ^^^
 [ecj-lint] Resource leak: 'req' is never closed
 [ecj-lint] ----------
 [ecj-lint] 30 problems (1 error, 29 warnings)

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2081: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2120: Compile failed; see the compiler error output for details.

Total time: 96 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1454 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1454/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

6 tests failed.
FAILED:  org.apache.solr.cloud.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42976 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42976 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E:798F6395C14A4E79]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:232)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:195)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:122)
	at org.apache.solr.cloud.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap(CdcrBootstrapTest.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42976 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
	... 46 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrBootstrapTest:     1) Thread[id=27281, name=Thread-6704, state=WAITING, group=TGRP-CdcrBootstrapTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)    2) Thread[id=27285, name=ProcessThread(sid:0 cport:42976):, state=WAITING, group=TGRP-CdcrBootstrapTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)    3) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=27283, name=SessionTracker, state=TIMED_WAITING, group=TGRP-CdcrBootstrapTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrBootstrapTest: 
   1) Thread[id=27281, name=Thread-6704, state=WAITING, group=TGRP-CdcrBootstrapTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   2) Thread[id=27285, name=ProcessThread(sid:0 cport:42976):, state=WAITING, group=TGRP-CdcrBootstrapTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   3) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=27283, name=SessionTracker, state=TIMED_WAITING, group=TGRP-CdcrBootstrapTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]         at sun.nio.ch.IOUtil.drain(Native Method)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:109)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
        at sun.nio.ch.IOUtil.drain(Native Method)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:109)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=22317, name=jetty-launcher-4213-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=22318, name=jetty-launcher-4213-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=22317, name=jetty-launcher-4213-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=22318, name=jetty-launcher-4213-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication

Error Message:
Index: 0, Size: 0

Stack Trace:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E:BA1017B756126B20]:0)
	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
	at java.util.ArrayList.get(ArrayList.java:429)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication(TestReplicationHandler.java:561)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry

Error Message:
Captured an uncaught exception in thread: Thread[id=18440, name=OverseerAutoScalingTriggerThread-98772829996646402-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=18440, name=OverseerAutoScalingTriggerThread-98772829996646402-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E:78876DB84308C5A6]:0)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
	... 5 more




Build Log:
[...truncated 12129 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> 1500520 INFO  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/init-core-data-001
   [junit4]   2> 1500521 WARN  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 1500523 INFO  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1500525 INFO  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1500542 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestReplicateAfterCoreReload
   [junit4]   2> 1500542 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1
   [junit4]   2> 1500546 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1500547 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@a332755{/solr,null,AVAILABLE}
   [junit4]   2> 1500557 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a6274a5{HTTP/1.1,[http/1.1]}{127.0.0.1:57561}
   [junit4]   2> 1500559 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1502772ms
   [junit4]   2> 1500559 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=57561}
   [junit4]   2> 1500559 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.482Z
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/solr.xml
   [junit4]   2> 1500563 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1500563 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1500564 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1500609 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500615 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500615 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500617 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/.
   [junit4]   2> 1500617 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1500617 INFO  (coreLoadExecutor-2937-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1500633 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1500640 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1500643 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1500699 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1500699 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1500700 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1500700 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@52354db4[collection1] main]
   [junit4]   2> 1500700 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1500701 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1500701 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1500701 INFO  (searcherExecutor-2938-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@52354db4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1500702 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1
   [junit4]   2> 1500704 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6cd8f708{/solr,null,AVAILABLE}
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d2a2dbd{HTTP/1.1,[http/1.1]}{127.0.0.1:41938}
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1502919ms
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=41938}
   [junit4]   2> 1500706 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.631Z
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/solr.xml
   [junit4]   2> 1500711 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1500712 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1500712 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1500743 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500749 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500750 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500751 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/.
   [junit4]   2> 1500751 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1500752 INFO  (coreLoadExecutor-2947-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1500765 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1500770 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1500773 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1500808 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1500808 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1500809 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1500809 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7e3954ec[collection1] main]
   [junit4]   2> 1500809 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/conf
   [junit4]   2> 1500810 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1500810 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1500822 INFO  (searcherExecutor-2948-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7e3954ec[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1500822 INFO  (qtp353127954-13418) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@3d13fd3a commitCommandVersion:0
   [junit4]   2> 1500827 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6f409031[collection1] main]
   [junit4]   2> 1500827 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1500828 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4d2a2dbd{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1500828 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=466123618
   [junit4]   2> 1500828 INFO  (searcherExecutor-2948-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6f409031[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1500828 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1500829 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2263b1e3: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@7f249039
   [junit4]   2> 1500834 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1500834 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@72ddb532: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5dbd2814
   [junit4]   2> 1500837 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1500837 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@60882d81: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@17a91bcc
   [junit4]   2> 1500841 INFO  (coreCloseExecutor-2953-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@28807124
   [junit4]   2> 1500841 INFO  (coreCloseExecutor-2953-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=679506212
   [junit4]   2> 1500841 INFO  (coreCloseExecutor-2953-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1b05db3d: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@3a9213b4
   [junit4]   2> 1500857 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6cd8f708{/solr,null,UNAVAILABLE}
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@a6274a5{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1822009639
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cc7c730: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@4818923d
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1500861 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@661bd96c: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5dbd2814
   [junit4]   2> 1500861 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1500861 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4e89ecd4: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@17a91bcc
   [junit4]   2> 1500864 INFO  (coreCloseExecutor-2955-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@73905b03
   [junit4]   2> 1500864 INFO  (coreCloseExecutor-2955-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1938840323
   [junit4]   2> 1500864 INFO  (coreCloseExecutor-2955-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ad56924: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@7b685246
   [junit4]   2> 1500872 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@a332755{/solr,null,UNAVAILABLE}
   [junit4]   2> 1500875 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79931642{/solr,null,AVAILABLE}
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3aa7f59b{HTTP/1.1,[http/1.1]}{127.0.0.1:39666}
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1503089ms
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=39666}
   [junit4]   2> 1500877 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.801Z
   [junit4]   2> 1500880 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/solr.xml
   [junit4]   2> 1500883 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1500883 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1500884 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1500918 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500926 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500926 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500927 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/.
   [junit4]   2> 1500927 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1500928 INFO  (coreLoadExecutor-2962-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1500941 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1500947 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1500949 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1501005 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1501005 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@420607b0[collection1] main]
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1501006 INFO  (searcherExecutor-2963-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@420607b0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501017 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1501018 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 0
   [junit4]   2> 1501018 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1501018 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@214a24f6 commitCommandVersion:0
   [junit4]   2> 1501023 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6eb0d494[collection1] main]
   [junit4]   2> 1501024 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1501025 INFO  (searcherExecutor-2963-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6eb0d494[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501025 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 6
   [junit4]   2> 1501026 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501026 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 0 docs
   [junit4]   2> 1501030 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1501031 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@78a375e2{/solr,null,AVAILABLE}
   [junit4]   2> 1501031 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@9d32c0d{HTTP/1.1,[http/1.1]}{127.0.0.1:49567}
   [junit4]   2> 1501031 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1503244ms
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=49567}
   [junit4]   2> 1501032 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 8.0.0
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.954Z
   [junit4]   2> 1501035 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/solr.xml
   [junit4]   2> 1501038 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1501038 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1501039 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1501081 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501090 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501090 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501091 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/.
   [junit4]   2> 1501091 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1501093 INFO  (coreLoadExecutor-2972-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1501108 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1501114 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1501115 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1501115 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1501116 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501116 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1501116 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1501117 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1501156 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1501156 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1501157 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1501157 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@111292d9[collection1] main]
   [junit4]   2> 1501157 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/conf
   [junit4]   2> 1501158 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1501158 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1501159 INFO  (searcherExecutor-2973-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@111292d9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501173 INFO  (qtp1265301787-13477) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501173 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 0 docs
   [junit4]   2> 1501174 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getIndexVersion&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1501174 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getCommits&qt=/replication&wt=javabin&version=2&command=commits} status=0 QTime=0
   [junit4]   2> 1501187 INFO  (qtp863784252-13457) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1501192 INFO  (qtp863784252-13457) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1501194 INFO  (qtp863784252-13457) [    ] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1501194 INFO  (qtp863784252-13457) [    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1
   [junit4]   2> 1501212 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501212 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1501254 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1501254 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1501255 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2685b25e[collection1] main]
   [junit4]   2> 1501255 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1501256 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1501256 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1501259 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=69.4169921875, floorSegmentMB=1.2412109375, forceMergeDeletesPctAllowed=8.19040782526387, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3784452969658093
   [junit4]   2> 1501259 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1501260 INFO  (searcherExecutor-2978-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2685b25e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@100af24[collection1] main]
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7332e061
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1932714081
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f1ba02f: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@689ace5a
   [junit4]   2> 1501261 INFO  (searcherExecutor-2978-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@100af24[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501265 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=collection1&qt=/admin/cores&action=reload&wt=javabin&version=2} status=0 QTime=89
   [junit4]   2> 1501266 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getIndexVersion&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1501266 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getCommits&qt=/replication&wt=javabin&version=2&command=commits} status=0 QTime=0
   [junit4]   2> 1501267 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 1501270 INFO  (qtp863784252-13464) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 2
   [junit4]   2> 1501271 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1501271 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7c7e962c commitCommandVersion:0
   [junit4]   2> 1501272 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@cd5125f[collection1] main]
   [junit4]   2> 1501272 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1501273 INFO  (searcherExecutor-2978-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@cd5125f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C2)))}
   [junit4]   2> 1501273 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
   [junit4]   2> 1501273 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=2 status=0 QTime=0
   [junit4]   2> 1501274 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 2 docs
   [junit4]   2> 1501274 INFO  (qtp1265301787-13478) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501274 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501376 INFO  (qtp1265301787-13479) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501376 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501477 INFO  (qtp1265301787-13480) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501477 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501578 INFO  (qtp1265301787-13481) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501578 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501624 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1501624 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 3
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1507153690193
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1501627 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=1
   [junit4]   2> 1501627 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 10
   [junit4]   2> 1501630 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=229680935, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1501633 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1501633 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@5ad19eb2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79043286)
   [junit4]   2> 1501633 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501634 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501637 INFO  (qtp863784252-13464) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.tip&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501638 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501639 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501640 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.tim&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501640 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501641 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501642 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501642 INFO  (qtp863784252-13464) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=segments_3&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501643 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1503) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@5ad19eb2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79043286)
   [junit4]   2> 1501644 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=54.041015625, floorSegmentMB=0.8642578125, forceMergeDeletesPctAllowed=11.80472124987806, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1663439213008874
   [junit4]   2> 1501647 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1501647 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@9b5a984[collection1] main]
   [junit4]   2> 1501648 INFO  (searcherExecutor-2973-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@9b5a984[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C2)))}
   [junit4]   2> 1501679 INFO  (qtp1265301787-13482) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=2 status=0 QTime=0
   [junit4]   2> 1501679 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 400ms and found 2 docs
   [junit4]   2> 1501679 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending doTestReplicateAfterCoreReload
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3aa7f59b{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=980837206
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4db30fc6: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1e91ac1a
   [junit4]   2> 1501684 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1501684 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@220e3046: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5dbd2814
   [junit4]   2> 1501686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1501686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@446e2522: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@17a91bcc
   [junit4]   2> 1501688 INFO  (coreCloseExecutor-2983-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4056a276
   [junit4]   2> 1501688 INFO  (coreCloseExecutor-2983-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1079419510
   [junit4]   2> 1501688 INFO  (coreCloseExecutor-2983-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@601bf468: rootName = null, domain

[...truncated too long message...]

sUnloadDistributedZkTest, CollectionsAPIAsyncDistributedZkTest, AutoScalingHandlerTest, ZkCLITest, DistribDocExpirationUpdateProcessorTest, TestScoreJoinQPScore, TestSolrCoreSnapshots, SimplePostToolTest, TestXIncludeConfig, TestRecoveryHdfs, PrimitiveFieldTypeTest, TestConfigSetImmutable, RecoveryAfterSoftCommitTest, FastVectorHighlighterTest, TimeZoneUtilsTest, BlockJoinFacetRandomTest, OverseerTest, TestSlowCompositeReaderWrapper, TestQueryWrapperFilter, SolrInfoBeanTest, BufferStoreTest, SearchHandlerTest, RAMDirectoryFactoryTest, TestLFUCache, UniqFieldsUpdateProcessorFactoryTest, DistributedDebugComponentTest, TestFunctionQuery, TestRTimerTree, TestXmlQParserPlugin, SSLMigrationTest, TestManagedResourceStorage, AssignTest, TestCursorMarkWithoutUniqueKey, RecoveryZkTest, TestSha256AuthenticationProvider, ClusterStateUpdateTest, RestartWhileUpdatingTest, TestCloudRecovery, TestTestInjection, TestBinaryResponseWriter, CloudExitableDirectoryReaderTest, UUIDUpdateProcessorFallbackTest, CSVRequestHandlerTest, TestHalfAndHalfDocValues, TestPseudoReturnFields, BJQParserTest, CustomCollectionTest, InfixSuggestersTest, AnalysisAfterCoreReloadTest, ShardRoutingTest, ReplaceNodeTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestClassicSimilarityFactory, CoreAdminRequestStatusTest, DirectUpdateHandlerOptimizeTest, HdfsBasicDistributedZk2Test, TestLegacyFieldCache, TestUseDocValuesAsStored2, ExecutePlanActionTest, AtomicUpdateProcessorFactoryTest, TestSolrConfigHandlerConcurrent, TestTolerantSearch, SegmentsInfoRequestHandlerTest, SolrIndexSplitterTest, SyncSliceTest, ScriptEngineTest, SuggestComponentContextFilterQueryTest, AddBlockUpdateTest, MoveReplicaHDFSTest, DateFieldTest, TestRawTransformer, TestReplicationHandler, DistributedQueueTest, DistributedSuggestComponentTest, NodeLostTriggerTest, DocValuesTest, TestHashQParserPlugin, TestBinaryField, TestSweetSpotSimilarityFactory, TestWriterPerf, HdfsLockFactoryTest, TestLegacyNumericUtils, ParsingFieldUpdateProcessorsTest, SuggesterWFSTTest, UUIDFieldTest, TestSubQueryTransformerDistrib, TestCoreContainer, TestSolrCoreProperties, TestCloudPivotFacet, SolrCloudExampleTest, TestDFRSimilarityFactory, XmlUpdateRequestHandlerTest, SolrGraphiteReporterTest, SolrCoreTest, TestNoOpRegenerator, TestReversedWildcardFilterFactory, CurrencyFieldTypeTest, TestRestoreCore, GraphQueryTest, ClusterStateTest, DefaultValueUpdateProcessorTest, TestSchemaSimilarityResource, RequestHandlersTest, TestPolicyCloud, TestSolrDeletionPolicy2, TestSerializedLuceneMatchVersion, MultiThreadedOCPTest, HighlighterTest, TestDocBasedVersionConstraints, RulesTest, TestJmxIntegration, HighlighterConfigTest, TestRequestForwarding, SolrTestCaseJ4Test, TestOnReconnectListenerSupport, DistributedSpellCheckComponentTest, TestManagedStopFilterFactory, TestCodecSupport, TestRandomDVFaceting, TestStressRecovery, NotRequiredUniqueKeyTest, ResponseLogComponentTest, TestManagedSchemaAPI, TestSQLHandlerNonCloud, TestIntervalFaceting, TestStressCloudBlindAtomicUpdates, HealthCheckHandlerTest, DistributedFacetPivotLongTailTest, TestCustomSort, TestConfigSetsAPIExclusivity, TestDistributedStatsComponentCardinality, BasicAuthIntegrationTest, TestQuerySenderListener, TestMultiValuedNumericRangeQuery, MissingSegmentRecoveryTest, TestPhraseSuggestions, TestAuthenticationFramework, TestCloudInspectUtil, TestStressVersions, ChaosMonkeyNothingIsSafeTest, TestGraphMLResponseWriter, TestOrdValues, TestLockTree, QueryParsingTest, TestExportWriter, TestExclusionRuleCollectionAccess, DistributedFacetExistsSmallTest, TestBulkSchemaAPI, TestUpdate, TestSubQueryTransformer, OverseerTaskQueueTest, TestFreeTextSuggestions, TestSearcherReuse, DisMaxRequestHandlerTest, CollectionStateFormat2Test, MetricsConfigTest, SuggesterTest, TestReloadAndDeleteDocs, TestSolrCloudWithKerberosAlt, TestStressLiveNodes, TestIBSimilarityFactory, TestConfigsApi, OpenCloseCoreStressTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, ShardRoutingCustomTest, TestDistributedGrouping, TestFaceting, TestRecovery, TestHashPartitioner, TestRealTimeGet, TestJoin, HardAutoCommitTest, TestRangeQuery, ConvertedLegacyTest, TestSort, TestFiltering, TestLazyCores, ShowFileRequestHandlerTest, TestCoreDiscovery, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, TestStressLucene, TestTrie, SpatialFilterTest, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, DirectSolrSpellCheckerTest, TestOmitPositions, TermsComponentTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, IndexSchemaTest, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, TestComponentsName, SearchHandlerTest, TestDistributedMissingSort, ActionThrottleTest, AliasIntegrationTest, AsyncCallRequestStatusResponseTest, BaseCdcrDistributedZkTest, CdcrBootstrapTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CdcrBootstrapTest -Dtests.seed=AE584CE27515D63E -Dtests.slow=true -Dtests.locale=es-CU -Dtests.timezone=Africa/Sao_Tome -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | CdcrBootstrapTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrBootstrapTest: 
   [junit4]    >    1) Thread[id=27281, name=Thread-6704, state=WAITING, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1252)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1326)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]    >    2) Thread[id=27285, name=ProcessThread(sid:0 cport:42976):, state=WAITING, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]    >    3) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=27283, name=SessionTracker, state=TIMED_WAITING, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:109)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)
   [junit4] Completed [512/745 (4!)] on J0 in 122.33s, 3 tests, 3 errors <<< FAILURES!

[...truncated 47641 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:706: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:693: Source checkout is dirty (unversioned/missing files) after running tests!!! Offending files:
* lucene/licenses/morfologik-ukrainian-search-3.7.5.jar.sha1

Total time: 77 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any