You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/10/02 14:07:57 UTC

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1449 - Still Failing!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1449/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=30319, name=jetty-launcher-4309-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=30319, name=jetty-launcher-4309-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([8EBC4DF65222051A]:0)




Build Log:
[...truncated 13266 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8EBC4DF65222051A-001/init-core-data-001
   [junit4]   2> 3399209 WARN  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 3399209 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3399211 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 3399211 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3399251 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8EBC4DF65222051A-001/tempDir-001
   [junit4]   2> 3399251 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3399252 INFO  (Thread-8338) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3399252 INFO  (Thread-8338) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3399255 ERROR (Thread-8338) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3399353 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:54040
   [junit4]   2> 3399372 INFO  (jetty-launcher-4309-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3399372 INFO  (jetty-launcher-4309-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 3399374 INFO  (jetty-launcher-4309-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22999a72{/solr,null,AVAILABLE}
   [junit4]   2> 3399374 INFO  (jetty-launcher-4309-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@285c447d{/solr,null,AVAILABLE}
   [junit4]   2> 3399375 INFO  (jetty-launcher-4309-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@ad947b4{SSL,[ssl, http/1.1]}{127.0.0.1:39694}
   [junit4]   2> 3399375 INFO  (jetty-launcher-4309-thread-1) [    ] o.e.j.s.Server Started @3403109ms
   [junit4]   2> 3399375 INFO  (jetty-launcher-4309-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39694}
   [junit4]   2> 3399375 ERROR (jetty-launcher-4309-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3399375 INFO  (jetty-launcher-4309-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@79c82588{SSL,[ssl, http/1.1]}{127.0.0.1:61472}
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-2) [    ] o.e.j.s.Server Started @3403110ms
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61472}
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-02T13:40:05.658Z
   [junit4]   2> 3399376 ERROR (jetty-launcher-4309-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 3399376 INFO  (jetty-launcher-4309-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-02T13:40:05.658Z
   [junit4]   2> 3399378 INFO  (jetty-launcher-4309-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3399382 INFO  (jetty-launcher-4309-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3399390 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15edd50ad1f0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3399394 INFO  (jetty-launcher-4309-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54040/solr
   [junit4]   2> 3399396 INFO  (jetty-launcher-4309-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54040/solr
   [junit4]   2> 3399438 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3399439 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39694_solr
   [junit4]   2> 3399440 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.Overseer Overseer (id=98759580420341766-127.0.0.1:39694_solr-n_0000000000) starting
   [junit4]   2> 3399455 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39694_solr
   [junit4]   2> 3399457 INFO  (zkCallback-4321-thread-1-processing-n:127.0.0.1:39694_solr) [n:127.0.0.1:39694_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3399594 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3399597 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3399599 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61472_solr
   [junit4]   2> 3399601 INFO  (zkCallback-4320-thread-1-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3399601 INFO  (zkCallback-4321-thread-1-processing-n:127.0.0.1:39694_solr) [n:127.0.0.1:39694_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3399630 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 3399631 WARN  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 3399634 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 3399682 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 3399683 WARN  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 3399683 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 3399726 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 3399726 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 3399767 INFO  (jetty-launcher-4309-thread-2-EventThread) [n:127.0.0.1:61472_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 3399768 INFO  (jetty-launcher-4309-thread-1-EventThread) [n:127.0.0.1:39694_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 3399938 WARN  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 3399943 WARN  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
   [junit4]   2> 3400007 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61472.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ef8a2f6
   [junit4]   2> 3400011 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39694.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ef8a2f6
   [junit4]   2> 3400013 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61472.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ef8a2f6
   [junit4]   2> 3400013 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61472.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ef8a2f6
   [junit4]   2> 3400014 INFO  (jetty-launcher-4309-thread-2) [n:127.0.0.1:61472_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8EBC4DF65222051A-001/tempDir-001/node2/.
   [junit4]   2> 3400024 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39694.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ef8a2f6
   [junit4]   2> 3400024 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39694.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ef8a2f6
   [junit4]   2> 3400026 INFO  (jetty-launcher-4309-thread-1) [n:127.0.0.1:39694_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8EBC4DF65222051A-001/tempDir-001/node1/.
   [junit4]   2> 3400212 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 3400258 INFO  (qtp444197900-30280) [n:127.0.0.1:61472_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3400258 INFO  (qtp444197900-30280) [n:127.0.0.1:61472_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3400259 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4]   2> 3400442 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 3400486 INFO  (qtp444197900-30274) [n:127.0.0.1:61472_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3400486 INFO  (qtp444197900-30274) [n:127.0.0.1:61472_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3400486 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4]   2> 3400643 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 3400660 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 3400867 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testForwarding
   [junit4]   2> 3401473 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[8EBC4DF65222051A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3401475 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[8EBC4DF65222051A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54040/solr ready
   [junit4]   2> 3401510 INFO  (qtp449375795-30287) [n:127.0.0.1:39694_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3401514 INFO  (OverseerThreadFactory-11650-thread-1-processing-n:127.0.0.1:39694_solr) [n:127.0.0.1:39694_solr    ] o.a.s.c.CreateCollectionCmd Create collection forwardingCollection
   [junit4]   2> 3401626 INFO  (OverseerStateUpdate-98759580420341766-127.0.0.1:39694_solr-n_0000000000) [n:127.0.0.1:39694_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"forwardingCollection",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"forwardingCollection_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:61472/solr",
   [junit4]   2>   "type":"NRT"} 
   [junit4]   2> 3401839 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3401840 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3401947 INFO  (zkCallback-4320-thread-1-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3401947 INFO  (zkCallback-4320-thread-2-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3402867 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 3402900 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema [forwardingCollection_shard1_replica_n1] Schema name=test
   [junit4]   2> 3403031 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3403058 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'forwardingCollection_shard1_replica_n1' using configuration from collection forwardingCollection, trusted=true
   [junit4]   2> 3403059 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61472.solr.core.forwardingCollection.shard1.replica_n1' (registry 'solr.core.forwardingCollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ef8a2f6
   [junit4]   2> 3403059 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 3403059 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [[forwardingCollection_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8EBC4DF65222051A-001/tempDir-001/node2/forwardingCollection_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_8EBC4DF65222051A-001/tempDir-001/node2/./forwardingCollection_shard1_replica_n1/data/]
   [junit4]   2> 3403062 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4927537390059892]
   [junit4]   2> 3403145 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3403145 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3403147 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3403147 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3403150 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=888736960, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 3403151 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b7f7012[forwardingCollection_shard1_replica_n1] main]
   [junit4]   2> 3403153 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 3403157 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 3403175 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 3403176 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 3403176 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1580153290839031808
   [junit4]   2> 3403183 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 3403184 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 3403184 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 3403186 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 3403187 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 3403188 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 3403189 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 3403190 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 3403192 INFO  (searcherExecutor-11655-thread-1-processing-n:127.0.0.1:61472_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] Registered new searcher Searcher@4b7f7012[forwardingCollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3403194 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3403195 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3403195 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:61472/solr/forwardingCollection_shard1_replica_n1/
   [junit4]   2> 3403195 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3403195 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:61472/solr/forwardingCollection_shard1_replica_n1/ has no replicas
   [junit4]   2> 3403196 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3403203 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:61472/solr/forwardingCollection_shard1_replica_n1/ shard1
   [junit4]   2> 3403307 INFO  (zkCallback-4320-thread-1-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3403307 INFO  (zkCallback-4320-thread-2-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3403355 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3403358 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1519
   [junit4]   2> 3403367 INFO  (qtp449375795-30287) [n:127.0.0.1:39694_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 3403460 INFO  (zkCallback-4320-thread-2-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3403460 INFO  (zkCallback-4320-thread-1-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3403519 INFO  (OverseerCollectionConfigSetProcessor-98759580420341766-127.0.0.1:39694_solr-n_0000000000) [n:127.0.0.1:39694_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 3404370 INFO  (qtp449375795-30287) [n:127.0.0.1:39694_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=2860
   [junit4]   2> 3404370 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[8EBC4DF65222051A]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: forwardingCollection failOnTimeout:true timeout (sec):100
   [junit4]   2> 3404371 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[8EBC4DF65222051A]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: forwardingCollection
   [junit4]   2> 3404376 INFO  (qtp444197900-30274) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3404389 INFO  (qtp449375795-30289) [n:127.0.0.1:39694_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3404393 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 3404394 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testForwarding
   [junit4]   2> 3404584 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 3409665 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 3409907 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 3409909 INFO  (qtp444197900-30280) [n:127.0.0.1:61472_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3409909 INFO  (qtp444197900-30280) [n:127.0.0.1:61472_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3409910 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 3410171 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 3410174 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 3410441 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 3410446 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 3410725 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 3410767 WARN  (qtp444197900-30279) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
   [junit4]   2> id: invalid user name: "bar"
   [junit4]   2> 
   [junit4]   2> 3410781 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 3411028 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
   [junit4]   2> 3427802 WARN  (jetty-launcher-4309-thread-2-SendThread(127.0.0.1:54040)) [n:127.0.0.1:61472_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 17673ms for sessionid 0x15edd50ad1f0004
   [junit4]   2> 3427803 WARN  (jetty-launcher-4309-thread-2-SendThread(127.0.0.1:54040)) [n:127.0.0.1:61472_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 18165ms for sessionid 0x15edd50ad1f0007
   [junit4]   2> 3427803 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15edd50ad1f0004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3427802 WARN  (jetty-launcher-4309-thread-1-SendThread(127.0.0.1:54040)) [n:127.0.0.1:39694_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 18157ms for sessionid 0x15edd50ad1f0008
   [junit4]   2> 3427804 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15edd50ad1f0007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3427804 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15edd50ad1f0008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3427804 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
   [junit4]   2> 3427803 WARN  (jetty-launcher-4309-thread-1-SendThread(127.0.0.1:54040)) [n:127.0.0.1:39694_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 18158ms for sessionid 0x15edd50ad1f0006
   [junit4]   2> 3427805 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15edd50ad1f0006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3427915 INFO  (jetty-launcher-4309-thread-2-EventThread) [n:127.0.0.1:61472_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 3427917 WARN  (zkCallback-4320-thread-2-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50ba4a78 name: ZooKeeperConnection Watcher:127.0.0.1:54040/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 3427927 WARN  (zkCallback-4320-thread-2-processing-n:127.0.0.1:61472_solr) [n:127.0.0.1:61472_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 3427927 WARN  (zkCallback-4328-thread-4) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1aa8cdd3 name: ZooKeeperConnection Watcher:127.0.0.1:54040/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 3427927 WARN  (zkCallback-4328-thread-4) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 3427953 WARN  (zkCallback-4321-thread-5-processing-n:127.0.0.1:39694_solr) [n:127.0.0.1:39694_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@59b4358 name: ZooKeeperConnection Watcher:127.0.0.1:54040/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 3427953 WARN  (zkCallback-4321-thread-5-processing-n:127.0.0.1:39694_solr) [n:127.0.0.1:39694_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 3427985 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 3427985 INFO  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 3427986 WARN  (qtp444197900-30277) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.e.j.s.ServletHandler /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication, 
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	... 29 more
   [junit4]   2> 3427988 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4]   2> 3428206 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidRemote
   [junit4]   2> 3428214 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[8EBC4DF65222051A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidRemote
   [junit4]   2> 3428595 INFO  (jetty-closer-4310-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@ad947b4{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 3428598 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2070017018
   [junit4]   2> 3428598 INFO  (jetty-closer-4310-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@79c82588{SSL,[ssl, http/1.1]}{127.0.0.1:0}
   [junit4]   2> 3428598 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3428599 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@58a01694: rootName = solr_39694, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@252093a5
   [junit4]   2> 3428602 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=88969757
   [junit4]   2> 3428602 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 3428603 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4b4c4b6f: rootName = solr_61472, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1b5efcb0
   [junit4]   2> 3428612 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3428612 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@470f074b: rootName = solr_39694, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@4af369af
   [junit4]   2> 3428614 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 3428614 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@15066683: rootName = solr_61472, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@4af369af
   [junit4]   2> 3428619 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3428619 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@223d0c3c: rootName = solr_39694, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@50085238
   [junit4]   2> 3428619 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 3428619 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@26fe9308: rootName = solr_61472, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@50085238
   [junit4]   2> 3429483 INFO  (jetty-closer-4310-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 3429484 INFO  (coreCloseExecutor-11660-thread-1) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@22b76e9f
   [junit4]   2> 3429484 INFO  (coreCloseExecutor-11660-thread-1) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.forwardingCollection.shard1.replica_n1, tag=582446751
   [junit4]   2> 3429484 INFO  (coreCloseExecutor-11660-thread-1) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3e389e0e: rootName = solr_61472, domain = solr.core.forwardingCollection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.forwardingCollection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@e521c63
   [junit4]   2> 3429491 INFO  (coreCloseExecutor-11660-thread-1) [n:127.0.0.1:61472_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.forwardingCollection.shard1.leader, tag=582446751
   [junit4]   2> 3429494 ERROR (Thread[Thread-8355,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:61472_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 3429504 INFO  (jetty-closer-4310-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@22999a72{/solr,null,UNAVAILABLE}
   [junit4]   2> 3429856 WARN  (jetty-launcher-4309-thread-1-SendThread(127.0.0.1:54040)) [n:127.0.0.1:39694_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15edd50ad1f0008 has expired
   [junit4]   2> 3429874 INFO  (zkCallback-4321-thread-6-processing-n:127.0.0.1:39694_solr) [n:127.0.0.1:39694_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 3429875 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 3429876 INFO  (jetty-closer-4310-thread-2) [    ] o.a.s.c.Overseer Overseer (id=98759580420341766-127.0.0.1:39694_solr-n_0000000000) closing
   [junit4]   2> 3429877 INFO  (OverseerStateUpdate-98759580420341766-127.0.0.1:39694_solr-n_0000000000) [n:127.0.0.1:39694_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:39694_solr
   [junit4]   2> 3429880 WARN  (zkCallback-4321-thread-6-processing-n:127.0.0.1:39694_solr) [n:127.0.0.1:39694_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3429882 ERROR (Thread[Thread-8357,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:39694_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 3429892 INFO  (jetty-closer-4310-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@285c447d{/solr,null,UNAVAILABLE}
   [junit4]   2> 3429895 ERROR (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 3429896 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54040 54040
   [junit4]   2> 3439937 INFO  (Thread-8338) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54040 54040
   [junit4]   2> 3439941 WARN  (Thread-8338) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/collections/forwardingCollection/state.json
   [junit4]   2> 	4	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 3439970 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[8EBC4DF65222051A]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Oct 02, 2017 1:40:46 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 3458035 ERROR (jetty-launcher-4309-thread-2-EventThread) [n:127.0.0.1:61472_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.IllegalStateException: Client is not started
   [junit4]   2> 	at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
   [junit4]   2> 	at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:477)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceFacade.getZooKeeper(NamespaceFacade.java:112)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 02, 2017 1:41:06 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]   2>    1) Thread[id=30319, name=jetty-launcher-4309-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 02, 2017 1:41:06 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=30319, name=jetty-launcher-4309-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2> 3460142 ERROR (jetty-launcher-4309-thread-1-EventThread) [n:127.0.0.1:39694_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 3460143 WARN  (jetty-launcher-4309-thread-1-EventThread) [n:127.0.0.1:39694_solr    ] o.a.c.ConnectionState Session expired event received
   [junit4]   2> 3460145 WARN  (jetty-launcher-4309-thread-1-SendThread(127.0.0.1:54040)) [n:127.0.0.1:39694_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
   [junit4]   2> Oct 02, 2017 1:41:06 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=749, maxMBSortInHeap=6.7051487759892225, sim=RandomSimilarity(queryNorm=false): {}, locale=en-IN, timezone=America/Anguilla
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=3,free=179558920,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestPHPSerializedResponseWriter, AssignTest, TestFieldCacheSortRandom, EnumFieldTest, TestExactSharedStatsCache, TestCustomDocTransformer, ConjunctionSolrSpellCheckerTest, OverseerRolesTest, ConcurrentDeleteAndCreateCollectionTest, ScriptEngineTest, TestDefaultStatsCache, V2ApiIntegrationTest, ShardSplitTest, QueryParsingTest, SolrIndexMetricsTest, TestDFRSimilarityFactory, SolrCoreTest, TolerantUpdateProcessorTest, HdfsRecoverLeaseTest, UtilsToolTest, TestSolrCloudWithHadoopAuthPlugin, TestUnifiedSolrHighlighter, TestScoreJoinQPNoScore, TestCollectionAPI, TestInitParams, TestRetrieveFieldsOptimizer, TestDocTermOrds, TestReRankQParserPlugin, TestFileDictionaryLookup, TestFuzzyAnalyzedSuggestions, CursorMarkTest, TestDFISimilarityFactory, TestStressRecovery, TestRawResponseWriter, TestRandomCollapseQParserPlugin, TestManagedSchemaThreadSafety, DirectUpdateHandlerTest, TestIndexingPerformance, TestChildDocTransformer, DistributedExpandComponentTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, BinaryUpdateRequestHandlerTest, ReplicaListTransformerTest, TestReplicationHandler, DeleteStatusTest, TestMiniSolrCloudClusterSSL, SimpleMLTQParserTest, TestDownShardTolerantSearch, TestXIncludeConfig, AnalysisErrorHandlingTest, SortByFunctionTest, DefaultValueUpdateProcessorTest, CSVRequestHandlerTest, HdfsRestartWhileUpdatingTest, SolrCLIZkUtilsTest, TestWriterPerf, TestLFUCache, HighlighterConfigTest, StatsReloadRaceTest, ZkCLITest, TestJsonFacetRefinement, TestAuthorizationFramework, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SoftAutoCommitTest, TestGraphTermsQParserPlugin, BigEndianAscendingWordDeserializerTest, TlogReplayBufferedWhileIndexingTest, LeaderElectionTest, XsltUpdateRequestHandlerTest, TestSurroundQueryParser, TestLMDirichletSimilarityFactory, TestConfigSetsAPIZkFailure, SuggesterTest, TestSolrCoreProperties, TestUninvertingReader, RulesTest, TestCloudManagedSchema, SuggesterWFSTTest, TestPullReplicaErrorHandling, UniqFieldsUpdateProcessorFactoryTest, PreAnalyzedUpdateProcessorTest, CheckHdfsIndexTest, TestLazyCores, TestLocalFSCloudBackupRestore, SecurityConfHandlerTest, TemplateUpdateProcessorTest, TestSweetSpotSimilarityFactory, SearchHandlerTest, MultiThreadedOCPTest, TestCursorMarkWithoutUniqueKey, TestSolrCLIRunExample, AnalyticsMergeStrategyTest, TestFaceting, TestSubQueryTransformerCrossCore, RequiredFieldsTest, CollectionStateFormat2Test, TestPhraseSuggestions, TestHighFrequencyDictionaryFactory, TestReloadDeadlock, TestOmitPositions, BitVectorTest, TestPerFieldSimilarityWithDefaultOverride, TestCustomStream, TestSchemaResource, PathHierarchyTokenizerFactoryTest, ReplaceNodeTest, TestOverriddenPrefixQueryForCustomFieldType, CloneFieldUpdateProcessorFactoryTest, SimpleFacetsTest, CircularListTest, RAMDirectoryFactoryTest, IndexBasedSpellCheckerTest, MergeStrategyTest, DocValuesNotIndexedTest, TestRuleBasedAuthorizationPlugin, HdfsDirectoryFactoryTest, TestUtils, TestSQLHandlerNonCloud, TestFieldTypeResource, OverseerTest, TestReplicaProperties, TestFieldCacheReopen, TestJsonRequest, TestConfigSets, TestRestoreCore, AlternateDirectoryTest, TestTolerantSearch, TestExpandComponent, MinimalSchemaTest, SolrInfoBeanTest, AutoScalingHandlerTest, TestUpdate, TestStressUserVersions, HighlighterTest, TestCollectionAPIs, CoreAdminHandlerTest, SSLMigrationTest, MultiTermTest, DistanceUnitsTest, TestReloadAndDeleteDocs, TestNumericTerms32, TestHdfsCloudBackupRestore, TestQuerySenderNoQuery, TestFunctionQuery, BlockJoinFacetSimpleTest, TestPivotHelperCode, HdfsChaosMonkeyNothingIsSafeTest, TestLeaderElectionZkExpiry, FullHLLTest, NotRequiredUniqueKeyTest, DistributedFacetPivotSmallAdvancedTest, MissingSegmentRecoveryTest, TestSolrConfigHandlerConcurrent, UpdateLogTest, ClassificationUpdateProcessorIntegrationTest, ResponseLogComponentTest, DataDrivenBlockJoinTest, TestFoldingMultitermQuery, TimeZoneUtilsTest, SpatialHeatmapFacetsTest, SharedFSAutoReplicaFailoverUtilsTest, TestHashPartitioner, TestMinMaxOnMultiValuedField, TestSort, StressHdfsTest, OutputWriterTest, HdfsUnloadDistributedZkTest, TestGeoJSONResponseWriter, DeleteReplicaTest, ConfigureRecoveryStrategyTest, TestCollapseQParserPlugin, TestSimpleTextCodec, CursorPagingTest, TestMaxScoreQueryParser, SolrMetricReporterTest, TestXmlQParser, WordBreakSolrSpellCheckerTest, TestReplicationHandlerBackup, TestConfigReload, BasicDistributedZk2Test, UnloadDistributedZkTest, SyncSliceTest, LeaderElectionIntegrationTest, RecoveryZkTest, ClusterStateUpdateTest, TestRandomFaceting, ZkSolrClientTest, AutoCommitTest, TestDistributedGrouping, TestRecovery, DistributedSpellCheckComponentTest, TestRealTimeGet, TestJoin, HardAutoCommitTest, TestCoreContainer, TestSolr4Spatial, TestGroupingSearch, SolrCmdDistributorTest, QueryElevationComponentTest, PeerSyncTest, SignatureUpdateProcessorFactoryTest, DocValuesTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, TermVectorComponentTest, FastVectorHighlighterTest, RegexBoostProcessorTest, IndexSchemaTest, TestCollationField, ReturnFieldsTest, TestAnalyzedSuggestions, SOLR749Test, TestQuerySenderListener, CopyFieldTest, SolrIndexConfigTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, TestDocSet, TestBinaryField, NumericFieldsTest, TestElisionMultitermQuery, TestConfig, ExternalFileFieldSortTest, DirectSolrConnectionTest, TestCodecSupport, SynonymTokenizerTest, TestIBSimilarityFactory, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, DateMathParserTest, PrimUtilsTest, DateFieldTest, DOMUtilTest, ClusterStateTest, TestDocumentBuilder, SliceStateTest, UUIDFieldTest, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestLuceneIndexBackCompat, TestEmbeddedSolrServerAdminHandler, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, TestJettySolrRunner, ConnectionReuseTest, ActionThrottleTest, AliasIntegrationTest, AsyncCallRequestStatusResponseTest, BaseCdcrDistributedZkTest, CdcrBootstrapTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, CollectionsAPIAsyncDistributedZkTest, CustomCollectionTest, DistribDocExpirationUpdateProcessorTest, DistributedQueueTest, HealthCheckHandlerTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerTaskQueueTest, PeerSyncReplicationTest, RemoteQueryErrorTest, RestartWhileUpdatingTest, RollingRestartTest, SolrCloudExampleTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCloudPseudoReturnFields, TestCloudRecovery, TestClusterProperties, TestCollectionsAPIViaSolrCloudCluster, TestConfigSetsAPIExclusivity, TestDeleteCollectionOnDownNodes, TestDistribDocBasedVersion, TestLeaderElectionWithEmptyReplica, TestPrepRecovery, TestShortCircuitedRequests, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=8EBC4DF65222051A -Dtests.slow=true -Dtests.locale=en-IN -Dtests.timezone=America/Anguilla -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestSolrCloudWithSecureImpersonation (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]    >    1) Thread[id=30319, name=jetty-launcher-4309-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8EBC4DF65222051A]:0)
   [junit4] Completed [567/734 (1!)] on J0 in 60.95s, 12 tests, 1 error <<< FAILURES!

[...truncated 37784 lines...]
-documentation-lint:
     [echo] checking for broken html...
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [exec] 
     [exec] file:///build/docs/sandbox/org/apache/lucene/document/FloatPointNearestNeighbor.html
     [exec]   BROKEN LINK: file:///build/docs/core/org/apache/lucene/document/FloatPointNearestNeighbor.NearestHit.html
     [exec]   BROKEN LINK: file:///build/docs/core/org/apache/lucene/document/FloatPointNearestNeighbor.NearestHit.html
     [exec] 
     [exec] Broken javadocs links were found! Common root causes:
     [exec] * A typo of some sort for manually created links.
     [exec] * Public methods referencing non-public classes in their signature.

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build.xml:142: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build.xml:155: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2570: exec returned: 1

Total time: 100 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1456 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1456/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.SystemLogListenerTest

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:232)
	at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:190)
	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.setupCluster(SystemLogListenerTest.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
	... 30 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.SystemLogListenerTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest:     1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest: 
   1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.SystemLogListenerTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
        at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
        at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)




Build Log:
[...truncated 13224 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.SystemLogListenerTest
   [junit4]   2> 4011006 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_C11663E6AE8AF65E-001/init-core-data-001
   [junit4]   2> 4011007 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 4011009 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 4011010 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.SystemLogListenerTest_C11663E6AE8AF65E-001/tempDir-001
   [junit4]   2> 4011010 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4011011 INFO  (Thread-9801) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4011011 INFO  (Thread-9801) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4011017 ERROR (Thread-9801) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4011111 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64210
   [junit4]   2> 4041817 INFO  (SUITE-SystemLogListenerTest-seed#[C11663E6AE8AF65E]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Oct 05, 2017 7:38:13 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> Oct 05, 2017 7:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest: 
   [junit4]   2>    1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]   2>    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]   2>    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]   2>    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Oct 05, 2017 7:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]   2>    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]   2> 4062115 ERROR (ProcessThread(sid:0 cport:64210):) [    ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : ProcessThread(sid:0 cport:64210):
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]   2> 4062115 WARN  (Thread-9801) [    ] o.a.s.c.ZkTestServer Server interrupted
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]   2> 4062118 ERROR (ProcessThread(sid:0 cport:64210):) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 4062115 ERROR (SessionTracker) [    ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : SessionTracker
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]   2> 4062118 ERROR (SessionTracker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> Oct 05, 2017 7:38:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=ca-ES, timezone=Pacific/Guadalcanal
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=3,free=262362232,total=526385152
   [junit4]   2> NOTE: All tests run in this JVM: [TestReversedWildcardFilterFactory, CollectionStateFormat2Test, TestDocBasedVersionConstraints, TestSolrQueryResponse, TestMultiValuedNumericRangeQuery, TestFuzzyAnalyzedSuggestions, HdfsThreadLeakTest, OverseerCollectionConfigSetProcessorTest, MinimalSchemaTest, TestPullReplica, TestXmlQParserPlugin, MultiTermTest, AnalyticsQueryTest, SoftAutoCommitTest, TestIntervalFaceting, AssignTest, TestUpdate, OverseerRolesTest, LeaderInitiatedRecoveryOnShardRestartTest, UnloadDistributedZkTest, TestPseudoReturnFields, TestExclusionRuleCollectionAccess, ShufflingReplicaListTransformerTest, TestCustomStream, DistributedDebugComponentTest, SolrInfoBeanTest, ConfigureRecoveryStrategyTest, SolrMetricManagerTest, TestReplicationHandler, TestWriterPerf, TestManagedResourceStorage, CoreAdminRequestStatusTest, TestReloadAndDeleteDocs, DistribDocExpirationUpdateProcessorTest, SimplePostToolTest, SubstringBytesRefFilterTest, UUIDUpdateProcessorFallbackTest, CdcrReplicationDistributedZkTest, DeleteReplicaTest, AutoAddReplicasIntegrationTest, SolrCoreTest, TestQuerySenderNoQuery, TestPKIAuthenticationPlugin, MissingSegmentRecoveryTest, DirectUpdateHandlerTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, RulesTest, TestQuerySenderListener, TestBinaryField, TestFieldTypeCollectionResource, ConnectionReuseTest, HighlighterConfigTest, SortByFunctionTest, TestSolrCoreProperties, TestStressLiveNodes, BooleanFieldTest, QueryElevationComponentTest, OverseerTaskQueueTest, AlternateDirectoryTest, SuggestComponentContextFilterQueryTest, TestXIncludeConfig, InfoHandlerTest, AnalysisAfterCoreReloadTest, TestSolrXml, IndexSchemaTest, JSONWriterTest, TestGraphMLResponseWriter, CircularListTest, BaseCdcrDistributedZkTest, SolrJmxReporterTest, DirectUpdateHandlerOptimizeTest, TestStressVersions, DirectoryFactoryTest, TestSolrDeletionPolicy2, BlockCacheTest, DistributedQueryComponentCustomSortTest, SearchHandlerTest, TestImpersonationWithHadoopAuth, TestFieldCacheSort, TestOrdValues, TestRestoreCore, ZkControllerTest, BinaryUpdateRequestHandlerTest, TestEmbeddedSolrServerAdminHandler, TestReplicationHandlerBackup, DirectSolrConnectionTest, TestShortCircuitedRequests, DeleteLastCustomShardedReplicaTest, BigEndianAscendingWordSerializerTest, ExplicitHLLTest, TestDistributedStatsComponentCardinality, TestLockTree, ShardSplitTest, TestLeaderInitiatedRecoveryThread, SolrCloudExampleTest, FastVectorHighlighterTest, StressHdfsTest, TestSkipOverseerOperations, ReplaceNodeTest, DistributedFacetExistsSmallTest, TlogReplayBufferedWhileIndexingTest, DefaultValueUpdateProcessorTest, TestHighlightDedupGrouping, NotRequiredUniqueKeyTest, TestPHPSerializedResponseWriter, TestNamedUpdateProcessors, TestSort, TestPointFields, TestNonDefinedSimilarityFactory, CoreAdminHandlerTest, TestHdfsUpdateLog, TestBackupRepositoryFactory, TestFieldCollectionResource, TestRandomFlRTGCloud, TestIndexingPerformance, TestCoreContainer, ResponseHeaderTest, TestApiFramework, DistributedSpellCheckComponentTest, BasicDistributedZkTest, TestAddFieldRealTimeGet, LeaderFailureAfterFreshStartTest, CloudExitableDirectoryReaderTest, DistributedFacetPivotSmallTest, TestQueryWrapperFilter, LeaderElectionContextKeyTest, AddBlockUpdateTest, ConcurrentDeleteAndCreateCollectionTest, TestStressRecovery, ChaosMonkeyNothingIsSafeTest, XsltUpdateRequestHandlerTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestCloudPivotFacet, TestCloudRecovery, HdfsRecoverLeaseTest, TestPhraseSuggestions, FieldAnalysisRequestHandlerTest, TestTolerantUpdateProcessorCloud, TestRawTransformer, TestDistributedSearch, OutputWriterTest, TestExactStatsCache, RecoveryAfterSoftCommitTest, RegexBoostProcessorTest, RestartWhileUpdatingTest, TestHashQParserPlugin, DistanceFunctionTest, SSLMigrationTest, MoveReplicaHDFSTest, TestExtendedDismaxParser, TestRecoveryHdfs, TestFreeTextSuggestions, ReturnFieldsTest, NoCacheHeaderTest, SuggesterTest, TestLeaderElectionWithEmptyReplica, TestFieldCacheVsDocValues, ExecutePlanActionTest, LargeFieldTest, TestConfigReload, TestLegacyField, TestSearcherReuse, TestTestInjection, VMParamsZkACLAndCredentialsProvidersTest, RAMDirectoryFactoryTest, PrimitiveFieldTypeTest, TestFoldingMultitermQuery, HdfsRestartWhileUpdatingTest, QueryParsingTest, ClusterStateTest, TestSha256AuthenticationProvider, DisMaxRequestHandlerTest, TestDistribIDF, TestSurroundQueryParser, BasicAuthIntegrationTest, UniqFieldsUpdateProcessorFactoryTest, TestZkAclsWithHadoopAuth, ParsingFieldUpdateProcessorsTest, TestInPlaceUpdatesStandalone, TestRandomDVFaceting, ZkCLITest, TestFaceting, TestRecovery, TestRealTimeGet, TestStressReorder, TestJoin, DistributedTermsComponentTest, TestSolr4Spatial, SpellCheckComponentTest, TestGroupingSearch, PeerSyncTest, TestBadConfig, ShowFileRequestHandlerTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, DocValuesTest, DirectSolrSpellCheckerTest, TestOmitPositions, TermsComponentTest, TermVectorComponentTest, RequiredFieldsTest, SolrPluginUtilsTest, UpdateRequestProcessorFactoryTest, TestAnalyzedSuggestions, TestPartialUpdateDeduplication, TestLFUCache, TestLRUCache, TestDocumentBuilder, SliceStateTest, FileUtilsTest, CursorPagingTest, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestEmbeddedSolrServerSchemaAPI, AliasIntegrationTest, CdcrRequestHandlerTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CleanupOldIndexTest, ConnectionManagerTest, CreateCollectionCleanupTest, CustomCollectionTest, DeleteShardTest, DistribCursorPagingTest, DocValuesNotIndexedTest, HttpPartitionTest, MoveReplicaTest, NodeMutatorTest, RemoteQueryErrorTest, ReplicationFactorTest, RollingRestartTest, SimpleCollectionCreateDeleteTest, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestClusterProperties, TestCollectionsAPIViaSolrCloudCluster, TestConfigSetsAPIExclusivity, TestCryptoKeys, TestLocalFSCloudBackupRestore, TestOnReconnectListenerSupport, TestReplicaProperties, TestRequestForwarding, TestRequestStatusCollectionAPI, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressInPlaceUpdates, TestTolerantUpdateProcessorRandomCloud, ZkFailoverTest, HttpTriggerListenerTest, NodeLostTriggerTest, SystemLogListenerTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SystemLogListenerTest -Dtests.seed=C11663E6AE8AF65E -Dtests.slow=true -Dtests.locale=ca-ES -Dtests.timezone=Pacific/Guadalcanal -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | SystemLogListenerTest (suite) <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:232)
   [junit4]    > 	at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:190)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.SystemLogListenerTest.setupCluster(SystemLogListenerTest.java:82)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64210 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
   [junit4]    > 	... 30 moreThrowable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.SystemLogListenerTest: 
   [junit4]    >    1) Thread[id=35688, name=SessionTracker, state=TIMED_WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]    >    2) Thread[id=35686, name=Thread-9801, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1252)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1326)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]    >    3) Thread[id=35690, name=ProcessThread(sid:0 cport:64210):, state=WAITING, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]    >    4) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=35687, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-SystemLogListenerTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]    >         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C11663E6AE8AF65E]:0)
   [junit4] Completed [564/745 (1!)] on J0 in 54.14s, 0 tests, 3 errors <<< FAILURES!

[...truncated 47455 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:706: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:693: Source checkout is dirty (unversioned/missing files) after running tests!!! Offending files:
* lucene/licenses/morfologik-ukrainian-search-3.7.5.jar.sha1

Total time: 96 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1455 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1455/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:     1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([9A88CB42154A7069]:0)




Build Log:
[...truncated 12247 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth
   [junit4]   2> 1291131 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/init-core-data-001
   [junit4]   2> 1291131 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1291132 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1291165 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/tempDir-001
   [junit4]   2> 1291165 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1291165 INFO  (Thread-3219) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1291165 INFO  (Thread-3219) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1291165 ERROR (Thread-3219) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1291265 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:48693
   [junit4]   2> 1291293 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1291293 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1291296 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@438368ab{/solr,null,AVAILABLE}
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36bbe5e0{/solr,null,AVAILABLE}
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3f8642{HTTP/1.1,[http/1.1]}{127.0.0.1:49542}
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-1) [    ] o.e.j.s.Server Started @1293188ms
   [junit4]   2> 1291297 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49542}
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@41e4895d{HTTP/1.1,[http/1.1]}{127.0.0.1:48877}
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.e.j.s.Server Started @1293188ms
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=48877}
   [junit4]   2> 1291298 ERROR (jetty-launcher-2838-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1291298 ERROR (jetty-launcher-2838-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-05T08:29:40.581Z
   [junit4]   2> 1291298 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-05T08:29:40.581Z
   [junit4]   2> 1291306 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1291322 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1291325 INFO  (jetty-launcher-2838-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48693/solr
   [junit4]   2> 1291330 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00003, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1291333 INFO  (jetty-launcher-2838-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48693/solr
   [junit4]   2> 1291482 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1291484 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1291485 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:48877_solr
   [junit4]   2> 1291485 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49542_solr
   [junit4]   2> 1291486 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.Overseer Overseer (id=98775346737840134-127.0.0.1:48877_solr-n_0000000000) starting
   [junit4]   2> 1291487 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1291487 INFO  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1291496 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48877_solr
   [junit4]   2> 1291497 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1291498 INFO  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1291548 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 1291551 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=staff, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 1291555 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 1291572 INFO  (jetty-launcher-2838-thread-1-EventThread) [n:127.0.0.1:49542_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 1291573 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 1291574 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=staff, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=serv1-vm3,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
   [junit4]   2> 1291574 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 1291578 INFO  (jetty-launcher-2838-thread-2-EventThread) [n:127.0.0.1:48877_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 1291597 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 1291614 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 1291656 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48877.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291657 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49542.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291662 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48877.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291662 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_48877.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49542.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_49542.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67fe6531
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-2) [n:127.0.0.1:48877_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/tempDir-001/node2/.
   [junit4]   2> 1291663 INFO  (jetty-launcher-2838-thread-1) [n:127.0.0.1:49542_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_9A88CB42154A7069-001/tempDir-001/node1/.
   [junit4]   2> 1291702 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1291703 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48693/solr ready
   [junit4]   2> 1291856 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 1291877 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 1292009 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 1292014 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1292014 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1292014 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 1292145 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 1292160 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 1292300 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 1292302 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1292302 INFO  (qtp776626350-14678) [n:127.0.0.1:48877_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1292302 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4]   2> 1292433 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 1292457 WARN  (qtp776626350-14673) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
   [junit4]   2> id: invalid user name: "bar"
   [junit4]   2> 
   [junit4]   2> 1292461 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 1292612 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 1292614 INFO  (qtp776626350-14676) [n:127.0.0.1:48877_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1292614 INFO  (qtp776626350-14676) [n:127.0.0.1:48877_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1292614 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4]   2> 1292750 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 1292754 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 1292890 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 1307909 WARN  (jetty-launcher-2838-thread-1-SendThread(127.0.0.1:48693)) [n:127.0.0.1:49542_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16290ms for sessionid 0x15eeba78ee00007
   [junit4]   2> 1307910 WARN  (jetty-launcher-2838-thread-1-SendThread(127.0.0.1:48693)) [n:127.0.0.1:49542_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16245ms for sessionid 0x15eeba78ee00004
   [junit4]   2> 1307912 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 1307913 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1307913 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1307910 WARN  (jetty-launcher-2838-thread-2-SendThread(127.0.0.1:48693)) [n:127.0.0.1:48877_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16245ms for sessionid 0x15eeba78ee00006
   [junit4]   2> 1307909 WARN  (jetty-launcher-2838-thread-2-SendThread(127.0.0.1:48693)) [n:127.0.0.1:48877_solr    ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 16288ms for sessionid 0x15eeba78ee00008
   [junit4]   2> 1307913 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1307920 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15eeba78ee00008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1308011 WARN  (zkCallback-2856-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@739709c4 name: ZooKeeperConnection Watcher:127.0.0.1:48693/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1308012 WARN  (zkCallback-2856-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1308014 INFO  (jetty-launcher-2838-thread-1-EventThread) [n:127.0.0.1:49542_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 1308016 WARN  (zkCallback-2846-thread-3-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3fadfaa6 name: ZooKeeperConnection Watcher:127.0.0.1:48693/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1308016 WARN  (zkCallback-2846-thread-3-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1308022 WARN  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@b0fd95c name: ZooKeeperConnection Watcher:127.0.0.1:48693/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 1308022 WARN  (zkCallback-2850-thread-1-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 1308030 INFO  (jetty-launcher-2838-thread-2-EventThread) [n:127.0.0.1:48877_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 1308089 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 1308091 INFO  (qtp776626350-14665) [    ] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1308092 WARN  (qtp776626350-14665) [    ] o.e.j.s.ServletHandler /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication, 
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
   [junit4]   2> 	at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
   [junit4]   2> 	... 26 more
   [junit4]   2> 1308096 INFO  (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[9A88CB42154A7069]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4] IGNOR/A 0.00s J0 | TestImpersonationWithHadoopAuth.testForwarding
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/HADOOP-9893))
   [junit4]   2> 1308806 INFO  (jetty-closer-2839-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3f8642{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1308806 INFO  (jetty-closer-2839-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@41e4895d{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1308809 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1757477537
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1002081792
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@14768568: rootName = solr_48877, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@d58eca2
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1308811 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3e7445c: rootName = solr_49542, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@752a6817
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1cb50f68: rootName = solr_48877, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2853a7df
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1308822 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7a2bd70c: rootName = solr_49542, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@2853a7df
   [junit4]   2> 1308824 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1308824 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@51f0bb96: rootName = solr_48877, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4ee2e926
   [junit4]   2> 1308825 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1308825 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@779e0b35: rootName = solr_49542, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4ee2e926
   [junit4]   2> 1309247 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1309247 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1309248 INFO  (jetty-closer-2839-thread-1) [    ] o.a.s.c.Overseer Overseer (id=98775346737840134-127.0.0.1:48877_solr-n_0000000000) closing
   [junit4]   2> 1309248 INFO  (OverseerStateUpdate-98775346737840134-127.0.0.1:48877_solr-n_0000000000) [n:127.0.0.1:48877_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:48877_solr
   [junit4]   2> 1309248 INFO  (OverseerCollectionConfigSetProcessor-98775346737840134-127.0.0.1:48877_solr-n_0000000000) [n:127.0.0.1:48877_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=98775346737840134-127.0.0.1:48877_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 1309250 WARN  (OverseerAutoScalingTriggerThread-98775346737840134-127.0.0.1:48877_solr-n_0000000000) [n:127.0.0.1:48877_solr    ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 1309255 WARN  (zkCallback-2850-thread-5-processing-n:127.0.0.1:48877_solr) [n:127.0.0.1:48877_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1309256 ERROR (Thread[Thread-3238,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:48877_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 1309705 INFO  (jetty-closer-2839-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@36bbe5e0{/solr,null,UNAVAILABLE}
   [junit4]   2> 1309833 INFO  (zkCallback-2846-thread-4-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49542_solr
   [junit4]   2> 1309833 INFO  (zkCallback-2846-thread-5-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 1309834 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
   [junit4]   2> 1309834 INFO  (jetty-closer-2839-thread-2) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 1309836 WARN  (zkCallback-2846-thread-5-processing-n:127.0.0.1:49542_solr) [n:127.0.0.1:49542_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1311338 ERROR (Thread[Thread-3236,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:49542_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 1311345 INFO  (jetty-closer-2839-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@438368ab{/solr,null,UNAVAILABLE}
   [junit4]   2> 1311348 ERROR (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1311348 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48693 48693
   [junit4]   2> 1314684 INFO  (Thread-3219) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:48693 48693
   [junit4]   2> 1314690 WARN  (Thread-3219) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 	3	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 
   [junit4]   2> 1314690 INFO  (SUITE-TestImpersonationWithHadoopAuth-seed#[9A88CB42154A7069]-worker) [    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> Oct 05, 2017 8:30:03 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> Oct 05, 2017 8:30:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   [junit4]   2>    1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2>    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 05, 2017 8:30:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2>    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]   2> 1334818 ERROR (jetty-launcher-2838-thread-1-EventThread) [n:127.0.0.1:49542_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 1334818 ERROR (jetty-launcher-2838-thread-2-EventThread) [n:127.0.0.1:48877_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Oct 05, 2017 8:30:24 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-LY, timezone=America/Argentina/Jujuy
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=125433136,total=388169728
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsDirectoryFactoryTest, TestCustomStream, DistributedFacetPivotLongTailTest, TestPseudoReturnFields, TestSolrCloudWithKerberosAlt, TestStressVersions, TestQuerySenderListener, MissingSegmentRecoveryTest, MultiThreadedOCPTest, DistributedQueueTest, SimpleCollectionCreateDeleteTest, DateMathParserTest, SegmentsInfoRequestHandlerTest, TestSchemalessBufferedUpdates, TestGraphMLResponseWriter, ResourceLoaderTest, HardAutoCommitTest, TestSha256AuthenticationProvider, TestHdfsUpdateLog, CloudMLTQParserTest, BlockCacheTest, TestSolrQueryResponse, TestConfigSetImmutable, URLClassifyProcessorTest, ReplaceNodeTest, ZkCLITest, PeerSyncWithIndexFingerprintCachingTest, DistributedQueryComponentCustomSortTest, TestConfigReload, TestSmileRequest, DirectUpdateHandlerTest, TestFieldTypeCollectionResource, PeerSyncTest, TestBackupRepositoryFactory, SearchHandlerTest, CoreAdminHandlerTest, AutoScalingHandlerTest, TestLMDirichletSimilarityFactory, NoCacheHeaderTest, DistributedFacetPivotSmallTest, TimeZoneUtilsTest, TestCodecSupport, TestManagedSchemaAPI, AlternateDirectoryTest, TestInitParams, HdfsLockFactoryTest, GraphQueryTest, TestCSVResponseWriter, DisMaxRequestHandlerTest, CdcrRequestHandlerTest, TestLeaderElectionWithEmptyReplica, HealthCheckHandlerTest, TestSchemaNameResource, TestBinaryField, TestStressRecovery, ExecutePlanActionTest, ChaosMonkeyNothingIsSafeTest, CdcrReplicationDistributedZkTest, TestSolrCoreProperties, InfixSuggestersTest, MultiTermTest, TestAddFieldRealTimeGet, TestConfigSets, TestFieldCacheSort, LargeFieldTest, ConcurrentDeleteAndCreateCollectionTest, SimplePostToolTest, SubstringBytesRefFilterTest, ExplicitHLLTest, OutOfBoxZkACLAndCredentialsProvidersTest, AssignTest, TestSSLRandomization, TestFoldingMultitermQuery, UUIDFieldTest, TestIndexingPerformance, TestRandomDVFaceting, TestPHPSerializedResponseWriter, DistributedSpellCheckComponentTest, SpellCheckCollatorWithCollapseTest, TestNumericRangeQuery64, TestSolrCoreParser, TestCustomSort, LeaderFailureAfterFreshStartTest, ZkControllerTest, ResponseHeaderTest, TestMultiValuedNumericRangeQuery, TestReversedWildcardFilterFactory, TestFieldCollectionResource, SuggesterWFSTTest, TestCloudSchemaless, TestXIncludeConfig, TestBulkSchemaConcurrent, TestClusterProperties, TlogReplayBufferedWhileIndexingTest, SoftAutoCommitTest, MinimalSchemaTest, OverseerTest, TestFaceting, PathHierarchyTokenizerFactoryTest, SolrTestCaseJ4Test, ScriptEngineTest, DistributedFacetExistsSmallTest, ConnectionReuseTest, SolrCloudReportersTest, BasicAuthIntegrationTest, StressHdfsTest, TestConfigsApi, TestMacroExpander, ReturnFieldsTest, SSLMigrationTest, TestSkipOverseerOperations, ClusterStateTest, TestBlendedInfixSuggestions, SolrMetricsIntegrationTest, TestNamedUpdateProcessors, DocExpirationUpdateProcessorFactoryTest, RecoveryAfterSoftCommitTest, TestStressLiveNodes, TestIntervalFaceting, TestBulkSchemaAPI, TestPushWriter, TestNestedDocsSort, TestFieldCacheVsDocValues, ClassificationUpdateProcessorFactoryTest, SolrGraphiteReporterTest, NotRequiredUniqueKeyTest, TestLegacyNumericUtils, TestSimpleTextCodec, SolrIndexSplitterTest, TestPayloadCheckQParserPlugin, PKIAuthenticationIntegrationTest, FieldAnalysisRequestHandlerTest, SyncSliceTest, SolrCoreTest, TestBinaryResponseWriter, ClassificationUpdateProcessorTest, TestCloudInspectUtil, TestSolrCoreSnapshots, TestSubQueryTransformer, TestQuerySenderNoQuery, TestFunctionQuery, TestSQLHandlerNonCloud, WordBreakSolrSpellCheckerTest, WrapperMergePolicyFactoryTest, TestImpersonationWithHadoopAuth]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestImpersonationWithHadoopAuth -Dtests.seed=9A88CB42154A7069 -Dtests.slow=true -Dtests.locale=ar-LY -Dtests.timezone=America/Argentina/Jujuy -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestImpersonationWithHadoopAuth (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   [junit4]    >    1) Thread[id=14710, name=jetty-launcher-2838-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    >    2) Thread[id=14715, name=jetty-launcher-2838-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9A88CB42154A7069]:0)
   [junit4] Completed [263/745 (1!)] on J0 in 43.71s, 10 tests, 1 error, 1 skipped <<< FAILURES!

[...truncated 44597 lines...]
-ecj-javadoc-lint-src:
    [mkdir] Created dir: /var/tmp/ecj41246448
 [ecj-lint] Compiling 1149 source files to /var/tmp/ecj41246448
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. ERROR in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/cloud/autoscaling/ScheduledTriggers.java (at line 40)
 [ecj-lint] 	import java.util.concurrent.atomic.AtomicReference;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import java.util.concurrent.atomic.AtomicReference is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/CoreContainer.java (at line 1033)
 [ecj-lint] 	core = new SolrCore(this, dcore, coreConfig);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'core' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/HdfsDirectoryFactory.java (at line 234)
 [ecj-lint] 	dir = new BlockDirectory(path, hdfsDir, cache, null, blockCacheReadEnabled, false, cacheMerges, cacheReadOnce);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'dir' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 4. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 121)
 [ecj-lint] 	reader = cfiltfac.create(reader);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'reader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 145)
 [ecj-lint] 	return namedList;
 [ecj-lint] 	^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'listBasedTokenStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 6. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java (at line 1285)
 [ecj-lint] 	DirectoryReader reader = s==null ? null : s.get().getIndexReader();
 [ecj-lint] 	                ^^^^^^
 [ecj-lint] Resource leak: 'reader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 7. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 517)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zk, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] 8. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 743)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zkHost, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 9. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java (at line 578)
 [ecj-lint] 	tvWindowStream = new OffsetWindowTokenFilter(tvStream);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tvWindowStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/request/SimpleFacets.java (at line 925)
 [ecj-lint] 	fastForRandomSet = new HashDocSet(sset.getDocs(), 0, sset.size());
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fastForRandomSet' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java (at line 33)
 [ecj-lint] 	new SmileWriter(out, request, response).writeResponse();
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/schema/OpenExchangeRatesOrgProvider.java (at line 146)
 [ecj-lint] 	ratesJsonStream = resourceLoader.openResource(ratesFileLocation);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'ratesJsonStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java (at line 1315)
 [ecj-lint] 	UninvertingReader uninvertingReader = new UninvertingReader(new ReaderWrapper(searcher.getSlowAtomicReader(), collapseField), mapping);
 [ecj-lint] 	                  ^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'uninvertingReader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 145)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 154)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 16. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 163)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 17. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/OrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/ReverseOrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java (at line 291)
 [ecj-lint] 	throw new SolrException(ErrorCode.SERVER_ERROR, "Error occurred while loading solr.xml from zookeeper", e);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'loader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 20. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/CdcrTransactionLog.java (at line 98)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 191)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 261)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 24. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 25. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 260)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 26. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 275)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 27. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 529)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 28. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] 29. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 30. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SolrLogLayout.java (at line 149)
 [ecj-lint] 	SolrQueryRequest req = requestInfo == null ? null : requestInfo.getReq();
 [ecj-lint] 	                 ^^^
 [ecj-lint] Resource leak: 'req' is never closed
 [ecj-lint] ----------
 [ecj-lint] 30 problems (1 error, 29 warnings)

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2081: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2120: Compile failed; see the compiler error output for details.

Total time: 96 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1454 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1454/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

6 tests failed.
FAILED:  org.apache.solr.cloud.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42976 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42976 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E:798F6395C14A4E79]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:232)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:195)
	at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:122)
	at org.apache.solr.cloud.CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap(CdcrBootstrapTest.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42976 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
	... 46 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrBootstrapTest:     1) Thread[id=27281, name=Thread-6704, state=WAITING, group=TGRP-CdcrBootstrapTest]         at java.lang.Object.wait(Native Method)         at java.lang.Thread.join(Thread.java:1252)         at java.lang.Thread.join(Thread.java:1326)         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)    2) Thread[id=27285, name=ProcessThread(sid:0 cport:42976):, state=WAITING, group=TGRP-CdcrBootstrapTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)    3) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=27283, name=SessionTracker, state=TIMED_WAITING, group=TGRP-CdcrBootstrapTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrBootstrapTest: 
   1) Thread[id=27281, name=Thread-6704, state=WAITING, group=TGRP-CdcrBootstrapTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   2) Thread[id=27285, name=ProcessThread(sid:0 cport:42976):, state=WAITING, group=TGRP-CdcrBootstrapTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   3) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
        at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
        at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=27283, name=SessionTracker, state=TIMED_WAITING, group=TGRP-CdcrBootstrapTest]
        at java.lang.Object.wait(Native Method)
        at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]         at sun.nio.ch.IOUtil.drain(Native Method)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:109)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
        at sun.nio.ch.IOUtil.drain(Native Method)
        at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:109)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=22317, name=jetty-launcher-4213-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)    2) Thread[id=22318, name=jetty-launcher-4213-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=22317, name=jetty-launcher-4213-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   2) Thread[id=22318, name=jetty-launcher-4213-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication

Error Message:
Index: 0, Size: 0

Stack Trace:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E:BA1017B756126B20]:0)
	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
	at java.util.ArrayList.get(ArrayList.java:429)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication(TestReplicationHandler.java:561)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry

Error Message:
Captured an uncaught exception in thread: Thread[id=18440, name=OverseerAutoScalingTriggerThread-98772829996646402-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=18440, name=OverseerAutoScalingTriggerThread-98772829996646402-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E:78876DB84308C5A6]:0)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
	... 5 more




Build Log:
[...truncated 12129 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> 1500520 INFO  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/init-core-data-001
   [junit4]   2> 1500521 WARN  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
   [junit4]   2> 1500523 INFO  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1500525 INFO  (SUITE-TestReplicationHandler-seed#[AE584CE27515D63E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1500542 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestReplicateAfterCoreReload
   [junit4]   2> 1500542 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1
   [junit4]   2> 1500546 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1500547 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@a332755{/solr,null,AVAILABLE}
   [junit4]   2> 1500557 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@a6274a5{HTTP/1.1,[http/1.1]}{127.0.0.1:57561}
   [junit4]   2> 1500559 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1502772ms
   [junit4]   2> 1500559 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=57561}
   [junit4]   2> 1500559 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.482Z
   [junit4]   2> 1500560 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/solr.xml
   [junit4]   2> 1500563 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1500563 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1500564 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1500609 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500615 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500615 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500617 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/.
   [junit4]   2> 1500617 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1500617 INFO  (coreLoadExecutor-2937-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1500633 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1500640 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1500642 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1500643 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1500699 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1500699 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1500700 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1500700 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@52354db4[collection1] main]
   [junit4]   2> 1500700 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1500701 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1500701 INFO  (coreLoadExecutor-2937-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1500701 INFO  (searcherExecutor-2938-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@52354db4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1500702 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1
   [junit4]   2> 1500704 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6cd8f708{/solr,null,AVAILABLE}
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4d2a2dbd{HTTP/1.1,[http/1.1]}{127.0.0.1:41938}
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1502919ms
   [junit4]   2> 1500706 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=41938}
   [junit4]   2> 1500706 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.631Z
   [junit4]   2> 1500709 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/solr.xml
   [junit4]   2> 1500711 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1500712 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1500712 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1500743 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500749 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500750 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500751 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/.
   [junit4]   2> 1500751 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1500752 INFO  (coreLoadExecutor-2947-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1500765 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1500770 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1500772 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1500773 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1500808 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1500808 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1500809 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1500809 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7e3954ec[collection1] main]
   [junit4]   2> 1500809 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/conf
   [junit4]   2> 1500810 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1500810 INFO  (coreLoadExecutor-2947-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1500822 INFO  (searcherExecutor-2948-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7e3954ec[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1500822 INFO  (qtp353127954-13418) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1500822 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@3d13fd3a commitCommandVersion:0
   [junit4]   2> 1500827 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6f409031[collection1] main]
   [junit4]   2> 1500827 INFO  (indexFetcher-2952-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1500828 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4d2a2dbd{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1500828 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=466123618
   [junit4]   2> 1500828 INFO  (searcherExecutor-2948-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6f409031[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1500828 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1500829 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2263b1e3: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@7f249039
   [junit4]   2> 1500834 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1500834 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@72ddb532: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5dbd2814
   [junit4]   2> 1500837 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1500837 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@60882d81: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@17a91bcc
   [junit4]   2> 1500841 INFO  (coreCloseExecutor-2953-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@28807124
   [junit4]   2> 1500841 INFO  (coreCloseExecutor-2953-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=679506212
   [junit4]   2> 1500841 INFO  (coreCloseExecutor-2953-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1b05db3d: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@3a9213b4
   [junit4]   2> 1500857 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6cd8f708{/solr,null,UNAVAILABLE}
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@a6274a5{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1822009639
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@cc7c730: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@4818923d
   [junit4]   2> 1500860 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1500861 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@661bd96c: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5dbd2814
   [junit4]   2> 1500861 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1500861 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4e89ecd4: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@17a91bcc
   [junit4]   2> 1500864 INFO  (coreCloseExecutor-2955-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@73905b03
   [junit4]   2> 1500864 INFO  (coreCloseExecutor-2955-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1938840323
   [junit4]   2> 1500864 INFO  (coreCloseExecutor-2955-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ad56924: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@7b685246
   [junit4]   2> 1500872 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@a332755{/solr,null,UNAVAILABLE}
   [junit4]   2> 1500875 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79931642{/solr,null,AVAILABLE}
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3aa7f59b{HTTP/1.1,[http/1.1]}{127.0.0.1:39666}
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1503089ms
   [junit4]   2> 1500876 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=39666}
   [junit4]   2> 1500877 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1500879 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.801Z
   [junit4]   2> 1500880 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/solr.xml
   [junit4]   2> 1500883 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1500883 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1500884 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1500918 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500926 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500926 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500927 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/.
   [junit4]   2> 1500927 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1500928 INFO  (coreLoadExecutor-2962-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1500941 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1500947 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1500948 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1500949 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1501005 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1501005 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@420607b0[collection1] main]
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1501006 INFO  (coreLoadExecutor-2962-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1501006 INFO  (searcherExecutor-2963-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@420607b0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501017 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1501018 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 0
   [junit4]   2> 1501018 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1501018 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@214a24f6 commitCommandVersion:0
   [junit4]   2> 1501023 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6eb0d494[collection1] main]
   [junit4]   2> 1501024 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1501025 INFO  (searcherExecutor-2963-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6eb0d494[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501025 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 6
   [junit4]   2> 1501026 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501026 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 0 docs
   [junit4]   2> 1501030 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1501031 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@78a375e2{/solr,null,AVAILABLE}
   [junit4]   2> 1501031 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@9d32c0d{HTTP/1.1,[http/1.1]}{127.0.0.1:49567}
   [junit4]   2> 1501031 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.Server Started @1503244ms
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=49567}
   [junit4]   2> 1501032 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 8.0.0
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1501032 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-04T21:48:09.954Z
   [junit4]   2> 1501035 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/solr.xml
   [junit4]   2> 1501038 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 1501038 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1501039 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1501081 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501090 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501090 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501091 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/.
   [junit4]   2> 1501091 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1501093 INFO  (coreLoadExecutor-2972-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1501108 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1501114 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1501115 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1501115 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1501116 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501116 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1501116 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1501117 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.86328125, floorSegmentMB=0.9970703125, forceMergeDeletesPctAllowed=2.487897048783277, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35354646909200416
   [junit4]   2> 1501156 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1501156 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1501157 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=74.1171875, floorSegmentMB=0.7744140625, forceMergeDeletesPctAllowed=4.108848394977814, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35848579947479964
   [junit4]   2> 1501157 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@111292d9[collection1] main]
   [junit4]   2> 1501157 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-002/collection1/conf
   [junit4]   2> 1501158 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1501158 INFO  (coreLoadExecutor-2972-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1501159 INFO  (searcherExecutor-2973-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@111292d9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501173 INFO  (qtp1265301787-13477) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501173 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 0 docs
   [junit4]   2> 1501174 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getIndexVersion&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1501174 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getCommits&qt=/replication&wt=javabin&version=2&command=commits} status=0 QTime=0
   [junit4]   2> 1501187 INFO  (qtp863784252-13457) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1501192 INFO  (qtp863784252-13457) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1501194 INFO  (qtp863784252-13457) [    ] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1501194 INFO  (qtp863784252-13457) [    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1
   [junit4]   2> 1501212 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ab33cd
   [junit4]   2> 1501212 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1501254 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1501254 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1501255 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2685b25e[collection1] main]
   [junit4]   2> 1501255 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_AE584CE27515D63E-001/solr-instance-001/collection1/conf
   [junit4]   2> 1501256 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1501256 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1501259 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=69.4169921875, floorSegmentMB=1.2412109375, forceMergeDeletesPctAllowed=8.19040782526387, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3784452969658093
   [junit4]   2> 1501259 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1501260 INFO  (searcherExecutor-2978-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2685b25e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@100af24[collection1] main]
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7332e061
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1932714081
   [junit4]   2> 1501260 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@f1ba02f: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@689ace5a
   [junit4]   2> 1501261 INFO  (searcherExecutor-2978-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@100af24[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1501265 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=collection1&qt=/admin/cores&action=reload&wt=javabin&version=2} status=0 QTime=89
   [junit4]   2> 1501266 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getIndexVersion&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1501266 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getCommits&qt=/replication&wt=javabin&version=2&command=commits} status=0 QTime=0
   [junit4]   2> 1501267 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 1501270 INFO  (qtp863784252-13464) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 2
   [junit4]   2> 1501271 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1501271 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7c7e962c commitCommandVersion:0
   [junit4]   2> 1501272 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@cd5125f[collection1] main]
   [junit4]   2> 1501272 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1501273 INFO  (searcherExecutor-2978-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@cd5125f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C2)))}
   [junit4]   2> 1501273 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
   [junit4]   2> 1501273 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=2 status=0 QTime=0
   [junit4]   2> 1501274 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 2 docs
   [junit4]   2> 1501274 INFO  (qtp1265301787-13478) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501274 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501376 INFO  (qtp1265301787-13479) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501376 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501477 INFO  (qtp1265301787-13480) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501477 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501578 INFO  (qtp1265301787-13481) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1501578 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waiting for 2 docs
   [junit4]   2> 1501624 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1501624 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 3
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1507153690193
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1501625 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1501627 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=1
   [junit4]   2> 1501627 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 10
   [junit4]   2> 1501630 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=229680935, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1501633 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1501633 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@5ad19eb2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79043286)
   [junit4]   2> 1501633 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501634 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501637 INFO  (qtp863784252-13464) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.tip&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501638 INFO  (qtp863784252-13457) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501639 INFO  (qtp863784252-13459) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501640 INFO  (qtp863784252-13460) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.tim&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501640 INFO  (qtp863784252-13461) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501641 INFO  (qtp863784252-13462) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501642 INFO  (qtp863784252-13463) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501642 INFO  (qtp863784252-13464) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=segments_3&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1501643 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1503) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@5ad19eb2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79043286)
   [junit4]   2> 1501644 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=54.041015625, floorSegmentMB=0.8642578125, forceMergeDeletesPctAllowed=11.80472124987806, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1663439213008874
   [junit4]   2> 1501647 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1501647 INFO  (indexFetcher-2977-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@9b5a984[collection1] main]
   [junit4]   2> 1501648 INFO  (searcherExecutor-2973-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@9b5a984[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):C2)))}
   [junit4]   2> 1501679 INFO  (qtp1265301787-13482) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=2 status=0 QTime=0
   [junit4]   2> 1501679 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.h.TestReplicationHandler Waited for 400ms and found 2 docs
   [junit4]   2> 1501679 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending doTestReplicateAfterCoreReload
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3aa7f59b{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=980837206
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1501680 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4db30fc6: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1e91ac1a
   [junit4]   2> 1501684 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1501684 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@220e3046: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@5dbd2814
   [junit4]   2> 1501686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1501686 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[AE584CE27515D63E]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@446e2522: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@17a91bcc
   [junit4]   2> 1501688 INFO  (coreCloseExecutor-2983-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4056a276
   [junit4]   2> 1501688 INFO  (coreCloseExecutor-2983-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1079419510
   [junit4]   2> 1501688 INFO  (coreCloseExecutor-2983-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@601bf468: rootName = null, domain

[...truncated too long message...]

sUnloadDistributedZkTest, CollectionsAPIAsyncDistributedZkTest, AutoScalingHandlerTest, ZkCLITest, DistribDocExpirationUpdateProcessorTest, TestScoreJoinQPScore, TestSolrCoreSnapshots, SimplePostToolTest, TestXIncludeConfig, TestRecoveryHdfs, PrimitiveFieldTypeTest, TestConfigSetImmutable, RecoveryAfterSoftCommitTest, FastVectorHighlighterTest, TimeZoneUtilsTest, BlockJoinFacetRandomTest, OverseerTest, TestSlowCompositeReaderWrapper, TestQueryWrapperFilter, SolrInfoBeanTest, BufferStoreTest, SearchHandlerTest, RAMDirectoryFactoryTest, TestLFUCache, UniqFieldsUpdateProcessorFactoryTest, DistributedDebugComponentTest, TestFunctionQuery, TestRTimerTree, TestXmlQParserPlugin, SSLMigrationTest, TestManagedResourceStorage, AssignTest, TestCursorMarkWithoutUniqueKey, RecoveryZkTest, TestSha256AuthenticationProvider, ClusterStateUpdateTest, RestartWhileUpdatingTest, TestCloudRecovery, TestTestInjection, TestBinaryResponseWriter, CloudExitableDirectoryReaderTest, UUIDUpdateProcessorFallbackTest, CSVRequestHandlerTest, TestHalfAndHalfDocValues, TestPseudoReturnFields, BJQParserTest, CustomCollectionTest, InfixSuggestersTest, AnalysisAfterCoreReloadTest, ShardRoutingTest, ReplaceNodeTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestClassicSimilarityFactory, CoreAdminRequestStatusTest, DirectUpdateHandlerOptimizeTest, HdfsBasicDistributedZk2Test, TestLegacyFieldCache, TestUseDocValuesAsStored2, ExecutePlanActionTest, AtomicUpdateProcessorFactoryTest, TestSolrConfigHandlerConcurrent, TestTolerantSearch, SegmentsInfoRequestHandlerTest, SolrIndexSplitterTest, SyncSliceTest, ScriptEngineTest, SuggestComponentContextFilterQueryTest, AddBlockUpdateTest, MoveReplicaHDFSTest, DateFieldTest, TestRawTransformer, TestReplicationHandler, DistributedQueueTest, DistributedSuggestComponentTest, NodeLostTriggerTest, DocValuesTest, TestHashQParserPlugin, TestBinaryField, TestSweetSpotSimilarityFactory, TestWriterPerf, HdfsLockFactoryTest, TestLegacyNumericUtils, ParsingFieldUpdateProcessorsTest, SuggesterWFSTTest, UUIDFieldTest, TestSubQueryTransformerDistrib, TestCoreContainer, TestSolrCoreProperties, TestCloudPivotFacet, SolrCloudExampleTest, TestDFRSimilarityFactory, XmlUpdateRequestHandlerTest, SolrGraphiteReporterTest, SolrCoreTest, TestNoOpRegenerator, TestReversedWildcardFilterFactory, CurrencyFieldTypeTest, TestRestoreCore, GraphQueryTest, ClusterStateTest, DefaultValueUpdateProcessorTest, TestSchemaSimilarityResource, RequestHandlersTest, TestPolicyCloud, TestSolrDeletionPolicy2, TestSerializedLuceneMatchVersion, MultiThreadedOCPTest, HighlighterTest, TestDocBasedVersionConstraints, RulesTest, TestJmxIntegration, HighlighterConfigTest, TestRequestForwarding, SolrTestCaseJ4Test, TestOnReconnectListenerSupport, DistributedSpellCheckComponentTest, TestManagedStopFilterFactory, TestCodecSupport, TestRandomDVFaceting, TestStressRecovery, NotRequiredUniqueKeyTest, ResponseLogComponentTest, TestManagedSchemaAPI, TestSQLHandlerNonCloud, TestIntervalFaceting, TestStressCloudBlindAtomicUpdates, HealthCheckHandlerTest, DistributedFacetPivotLongTailTest, TestCustomSort, TestConfigSetsAPIExclusivity, TestDistributedStatsComponentCardinality, BasicAuthIntegrationTest, TestQuerySenderListener, TestMultiValuedNumericRangeQuery, MissingSegmentRecoveryTest, TestPhraseSuggestions, TestAuthenticationFramework, TestCloudInspectUtil, TestStressVersions, ChaosMonkeyNothingIsSafeTest, TestGraphMLResponseWriter, TestOrdValues, TestLockTree, QueryParsingTest, TestExportWriter, TestExclusionRuleCollectionAccess, DistributedFacetExistsSmallTest, TestBulkSchemaAPI, TestUpdate, TestSubQueryTransformer, OverseerTaskQueueTest, TestFreeTextSuggestions, TestSearcherReuse, DisMaxRequestHandlerTest, CollectionStateFormat2Test, MetricsConfigTest, SuggesterTest, TestReloadAndDeleteDocs, TestSolrCloudWithKerberosAlt, TestStressLiveNodes, TestIBSimilarityFactory, TestConfigsApi, OpenCloseCoreStressTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, ShardRoutingCustomTest, TestDistributedGrouping, TestFaceting, TestRecovery, TestHashPartitioner, TestRealTimeGet, TestJoin, HardAutoCommitTest, TestRangeQuery, ConvertedLegacyTest, TestSort, TestFiltering, TestLazyCores, ShowFileRequestHandlerTest, TestCoreDiscovery, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, TestStressLucene, TestTrie, SpatialFilterTest, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, DirectSolrSpellCheckerTest, TestOmitPositions, TermsComponentTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, IndexSchemaTest, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, TestComponentsName, SearchHandlerTest, TestDistributedMissingSort, ActionThrottleTest, AliasIntegrationTest, AsyncCallRequestStatusResponseTest, BaseCdcrDistributedZkTest, CdcrBootstrapTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CdcrBootstrapTest -Dtests.seed=AE584CE27515D63E -Dtests.slow=true -Dtests.locale=es-CU -Dtests.timezone=Africa/Sao_Tome -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | CdcrBootstrapTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrBootstrapTest: 
   [junit4]    >    1) Thread[id=27281, name=Thread-6704, state=WAITING, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1252)
   [junit4]    >         at java.lang.Thread.join(Thread.java:1326)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]    >    2) Thread[id=27285, name=ProcessThread(sid:0 cport:42976):, state=WAITING, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
   [junit4]    >    3) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    >    4) Thread[id=27283, name=SessionTracker, state=TIMED_WAITING, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=27282, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CdcrBootstrapTest]
   [junit4]    >         at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]    >         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:109)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
   [junit4]    >         at java.lang.Thread.run(Thread.java:748)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE584CE27515D63E]:0)
   [junit4] Completed [512/745 (4!)] on J0 in 122.33s, 3 tests, 3 errors <<< FAILURES!

[...truncated 47641 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:706: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:693: Source checkout is dirty (unversioned/missing files) after running tests!!! Offending files:
* lucene/licenses/morfologik-ukrainian-search-3.7.5.jar.sha1

Total time: 77 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1453 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1453/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry

Error Message:
Captured an uncaught exception in thread: Thread[id=25503, name=OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25503, name=OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]
	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893:E761CAB8C9D6AB0B]:0)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893]:0)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
	... 5 more




Build Log:
[...truncated 13277 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestLeaderElectionZkExpiry
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestLeaderElectionZkExpiry_31BEEBE2FFCBB893-001/init-core-data-001
   [junit4]   2> 2803362 WARN  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=5 numCloses=5
   [junit4]   2> 2803363 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 2803364 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2803364 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2803370 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLeaderElectionWithZkExpiry
   [junit4]   2> 2803373 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 2803419 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac
   [junit4]   2> 2803426 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac
   [junit4]   2> 2803426 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2569ac
   [junit4]   2> 2803427 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestLeaderElectionZkExpiry_31BEEBE2FFCBB893-001/testLeaderElectionWithZkExpiry-solr-001
   [junit4]   2> 2803428 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2803428 INFO  (Thread-5763) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2803428 INFO  (Thread-5763) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2803429 ERROR (Thread-5763) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2803528 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [    ] o.a.s.c.ZkTestServer start zk server on port:64902
   [junit4]   2> 2803659 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2803660 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
   [junit4]   2> 2803661 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530626-dummy.host.com:8984_solr-n_0000000000) starting
   [junit4]   2> 2803669 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
   [junit4]   2> 2803670 INFO  (zkCallback-4442-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2803674 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
   [junit4]   2> java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:182)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2803675 WARN  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Session 0x15ee72978f20002 for server 127.0.0.1/127.0.0.1:64902, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.io.IOException: Connection reset by peer
   [junit4]   2> 	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
   [junit4]   2> 	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
   [junit4]   2> 	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
   [junit4]   2> 2803776 WARN  (zkCallback-4442-thread-2-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2803776 WARN  (zkCallback-4442-thread-2-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2805232 WARN  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20002 has expired
   [junit4]   2> 2805232 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2805232 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2805232 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530626-dummy.host.com:8984_solr-n_0000000000) closing
   [junit4]   2> 2805232 WARN  (OverseerCollectionConfigSetProcessor-98770407157530626-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerTaskProcessor Overseer cannot talk to ZK
   [junit4]   2> 2805235 INFO  (OverseerStateUpdate-98770407157530626-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
   [junit4]   2> 2805237 ERROR (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:299)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2805276 WARN  (OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr    ] o.a.s.c.a.TriggerBase Exception checking ZK path /autoscaling/triggerState
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/triggerState
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:306)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:303)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:303)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:512)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:415)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.TriggerBase.<init>(TriggerBase.java:45)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.NodeLostTrigger.<init>(NodeLostTrigger.java:72)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScaling$TriggerFactory.create(AutoScaling.java:138)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.loadTriggers(OverseerTriggerThread.java:365)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.refreshAutoScalingConf(OverseerTriggerThread.java:312)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:137)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2805277 WARN  (zkCallback-4442-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> ???. 04, 2017 4:33:30 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000,5,Overseer autoscaling triggers]
   [junit4]   2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893]:0)
   [junit4]   2> 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
   [junit4]   2> 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2805288 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2805290 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2805290 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2805291 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 2805294 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x15ee72978f20003 due to java.nio.channels.CancelledKeyException
   [junit4]   2> 2805394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2805394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20003 has expired
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2806982 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2806983 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2806983 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2807082 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:898)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:475)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 2807083 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:357)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:354)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:898)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:475)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 10 more
   [junit4]   2> 
   [junit4]   2> 2807083 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2807085 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2807087 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2807088 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20004 has expired
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2808712 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2808714 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2808714 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2808813 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2808813 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2808813 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2808815 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2808822 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2808822 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20005 has expired
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2809871 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2809873 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2809873 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2809981 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2809981 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20006 has expired
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2811492 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2811493 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2811493 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2811592 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2811592 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2811592 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2811595 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2811603 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2811604 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2813394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20007 has expired
   [junit4]   2> 2813394 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2813395 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2813395 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2813397 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2813397 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2813495 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2813495 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2813496 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2813499 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2813499 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2813500 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-SendThread(127.0.0.1:64902)) [n:dummy.host.com:8984_solr    ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x15ee72978f20008 has expired
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33e50aa0 name: ZooKeeperConnection Watcher:127.0.0.1:64902/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2815184 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2815185 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2815185 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2815284 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2815284 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:322)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2815285 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2815287 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2815291 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr
   [junit4]   2> 2815293 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) starting
   [junit4]   2> 2815294 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.TestLeaderElectionZkExpiry Time=1294496485260407 Overseer leader is = dummy.host.com:8984_solr
   [junit4]   2> 2815299 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr
   [junit4]   2> 2815299 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) closing
   [junit4]   2> 2815300 INFO  (OverseerStateUpdate-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer According to ZK I (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 2815302 INFO  (OverseerStateUpdate-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr
   [junit4]   2> 2815302 INFO  (OverseerCollectionConfigSetProcessor-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=98770407157530634-dummy.host.com:8984_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 2815304 ERROR (OverseerCollectionConfigSetProcessor-98770407157530634-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr    ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer 
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Object.wait(Object.java:502)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:61)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:178)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2815307 WARN  (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer I'm exiting, but I'm still the leader
   [junit4]   2> 2815307 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 2815307 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2815308 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ee72978f2000a, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2816808 WARN  (zkCallback-4442-thread-1-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2816808 WARN  (zkCallback-4442-thread-6-processing-n:dummy.host.com:8984_solr) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2816809 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> 
   [junit4]   2> 2816810 ERROR (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:407)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:347)
   [junit4]   2> 	... 5 more
   [junit4]   2> 
   [junit4]   2> 2816810 WARN  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2816808 ERROR (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Could not delete my leader node 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:246)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:243)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:243)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:310)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 2816814 INFO  (zkCallback-4442-thread-5-processing-n:dummy.host.com:8984_solr-EventThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2816814 WARN  (OverseerExitThread) [n:dummy.host.com:8984_solr    ] o.a.s.c.Overseer Unable to rejoinElection 
   [junit4]   2> org.apache.solr.common.SolrException: Unable to rejoin election
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:1924)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:324)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect
   [junit4]   2> 	at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:772)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerElectionContext.copy(ElectionContext.java:815)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:385)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:1921)
   [junit4]   2> 	... 2 more
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerElectionContext.<init>(ElectionContext.java:770)
   [junit4]   2> 	... 5 more
   [junit4]   2> 2816814 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=959856624
   [junit4]   2> 2816815 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2816815 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@892c757: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5b81e7bc
   [junit4]   2> 2816818 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2816818 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@42733cb6: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@49ffb006
   [junit4]   2> 2816821 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2816821 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@50c718ed: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@5aa407f0
   [junit4]   2> 2816821 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 2816821 ERROR (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2816822 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64902 64902
   [junit4]   2> 2821884 INFO  (Thread-5763) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64902 64902
   [junit4]   2> 2821891 WARN  (Thread-5763) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 	4	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 2821892 INFO  (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[31BEEBE2FFCBB893]) [n:dummy.host.com:8984_solr    ] o.a.s.SolrTestCaseJ4 ###Ending testLeaderElectionWithZkExpiry
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLeaderElectionZkExpiry -Dtests.method=testLeaderElectionWithZkExpiry -Dtests.seed=31BEEBE2FFCBB893 -Dtests.slow=true -Dtests.locale=mk-MK -Dtests.timezone=Asia/Tashkent -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   18.5s J0 | TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25503, name=OverseerAutoScalingTriggerThread-98770407157530626-dummy.host.com:8984_solr-n_0000000000, state=RUNNABLE, group=Overseer autoscaling triggers]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893:E761CAB8C9D6AB0B]:0)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([31BEEBE2FFCBB893]:0)
   [junit4]    > 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:107)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:44)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers$ScheduledTrigger.<init>(ScheduledTriggers.java:398)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:149)
   [junit4]    > 	at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:220)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:78)
   [junit4]    > 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:105)
   [junit4]    > 	... 5 more
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=959856624
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 2821900 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 2821901 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 2821901 INFO  (SUITE-TestLeaderElectionZkExpiry-seed#[31BEEBE2FFCBB893]-worker) [n:dummy.host.com:8984_solr    ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=mk-MK, timezone=Asia/Tashkent
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=92765040,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestTolerantSearch, TestBinaryField, TestShortCircuitedRequests, TestCryptoKeys, DistributedDebugComponentTest, TestSolrCoreProperties, TestFreeTextSuggestions, TestSweetSpotSimilarityFactory, TestWriterPerf, HealthCheckHandlerTest, AnalyticsQueryTest, TestRestoreCore, MoveReplicaHDFSTest, TestCoreContainer, TestSolrQueryResponse, TestBackupRepositoryFactory, BooleanFieldTest, TestScoreJoinQPScore, SpatialFilterTest, TestUtils, TestLeaderInitiatedRecoveryThread, SortByFunctionTest, TestSlowCompositeReaderWrapper, BlockJoinFacetRandomTest, SSLMigrationTest, SolrCloudExampleTest, CdcrRequestHandlerTest, OverseerTest, TestRandomDVFaceting, PeerSyncTest, TestExportWriter, ResponseHeaderTest, TestLMDirichletSimilarityFactory, HdfsDirectoryFactoryTest, ScriptEngineTest, ConnectionReuseTest, TestSolrCloudWithKerberosAlt, TestDistributedSearch, CoreAdminRequestStatusTest, ConnectionManagerTest, TestSurroundQueryParser, AutoAddReplicasIntegrationTest, DirectSolrConnectionTest, RAMDirectoryFactoryTest, TestHttpShardHandlerFactory, V2ApiIntegrationTest, ClusterStateUpdateTest, TestAtomicUpdateErrorCases, UUIDFieldTest, TestUseDocValuesAsStored2, TestXIncludeConfig, TestFuzzyAnalyzedSuggestions, TestMissingGroups, DirectoryFactoryTest, DistributedFacetPivotSmallTest, TestExtendedDismaxParser, NumericFieldsTest, PKIAuthenticationIntegrationTest, TestSchemalessBufferedUpdates, DocValuesNotIndexedTest, TestBlendedInfixSuggestions, InfoHandlerTest, TestMacroExpander, VersionInfoTest, DistanceFunctionTest, TestSkipOverseerOperations, CurrencyFieldTypeTest, ResponseLogComponentTest, HdfsRecoverLeaseTest, ResourceLoaderTest, TestMaxTokenLenTokenizer, HLLSerializationTest, DeleteLastCustomShardedReplicaTest, TestConfigSetsAPIExclusivity, TestSha256AuthenticationProvider, TestNestedDocsSort, AtomicUpdateProcessorFactoryTest, TestXmlQParserPlugin, DeleteStatusTest, LeaderElectionTest, TestTolerantUpdateProcessorCloud, TestFieldCacheSort, TestEmbeddedSolrServerAdminHandler, TestRandomFlRTGCloud, CloudMLTQParserTest, TestSolrXml, TestCodecSupport, HdfsCollectionsAPIDistributedZkTest, DistributedSpellCheckComponentTest, TestBulkSchemaConcurrent, DistribDocExpirationUpdateProcessorTest, DefaultValueUpdateProcessorTest, TestNumericRangeQuery64, SliceStateTest, TestImpersonationWithHadoopAuth, TestSolrCoreParser, NodeLostTriggerTest, TestManagedResourceStorage, TlogReplayBufferedWhileIndexingTest, SegmentsInfoRequestHandlerTest, URLClassifyProcessorTest, TestConfigReload, GraphQueryTest, TestOrdValues, HttpTriggerListenerTest, TestSQLHandlerNonCloud, SuggesterWFSTTest, TestStandardQParsers, TestTestInjection, TestExclusionRuleCollectionAccess, SolrCoreTest, TestQuerySenderNoQuery, InfixSuggestersTest, TestReloadAndDeleteDocs, TestLFUCache, TestQuerySenderListener, TestUpdate, ChaosMonkeyNothingIsSafeTest, TestTrieFacet, AlternateDirectoryTest, HighlighterMaxOffsetTest, HdfsRestartWhileUpdatingTest, DirectUpdateHandlerOptimizeTest, LargeFieldTest, AssignTest, TestFieldTypeCollectionResource, ZkControllerTest, TestSubQueryTransformerDistrib, SolrMetricsIntegrationTest, TestAuthenticationFramework, TestExactStatsCache, BinaryUpdateRequestHandlerTest, TestConfigsApi, ExplicitHLLTest, SpellCheckCollatorWithCollapseTest, AddBlockUpdateTest, DistributedQueryComponentCustomSortTest, TestInPlaceUpdatesStandalone, TestHighlightDedupGrouping, TestRequestForwarding, SubstringBytesRefFilterTest, XmlUpdateRequestHandlerTest, XsltUpdateRequestHandlerTest, AutoScalingHandlerTest, HdfsThreadLeakTest, TestLegacyNumericUtils, RecoveryAfterSoftCommitTest, TestPushWriter, SolrMetricManagerTest, TestSubQueryTransformer, TestLegacyFieldCache, TestSerializedLuceneMatchVersion, TestLegacyNumericRangeQueryBuilder, TestReplicationHandlerBackup, AutoscalingHistoryHandlerTest, TestFilteredDocIdSet, TestPhraseSuggestions, DirectUpdateHandlerTest, TestSimpleTextCodec, DocValuesMissingTest, SparseHLLTest, LeaderInitiatedRecoveryOnShardRestartTest, TestFunctionQuery, SolrIndexSplitterTest, ZkStateReaderTest, PrimitiveFieldTypeTest, CollectionStateFormat2Test, TestNoOpRegenerator, CircularListTest, TestCustomStream, TestCursorMarkWithoutUniqueKey, TestOmitPositions, TestReversedWildcardFilterFactory, TestHdfsUpdateLog, SimpleCollectionCreateDeleteTest, TestManagedSynonymGraphFilterFactory, AliasIntegrationTest, SimplePostToolTest, BasicDistributedZkTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, ZkSolrClientTest, TestZkChroot, ZkCLITest, ShardRoutingCustomTest, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, HardAutoCommitTest, SimpleFacetsTest, SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, QueryElevationComponentTest, TestSort, TestLazyCores, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, TestSolrQueryParser, RegexBoostProcessorTest, TestJmxIntegration, ReturnFieldsTest, QueryParsingTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, HighlighterConfigTest, UpdateParamsTest, CopyFieldTest, SolrIndexConfigTest, TestStressRecovery, BaseCdcrDistributedZkTest, CdcrBootstrapTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CollectionsAPIAsyncDistributedZkTest, DeleteNodeTest, DeleteReplicaTest, DistribJoinFromCollectionTest, ForceLeaderTest, MissingSegmentRecoveryTest, MultiThreadedOCPTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, PeerSyncReplicationTest, SolrCLIZkUtilsTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudPivotFacet, TestCloudPseudoReturnFields, TestCloudRecovery, TestClusterProperties, TestCollectionsAPIViaSolrCloudCluster, TestDistribDocBasedVersion, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry]
   [junit4] Completed [547/745 (1!)] on J0 in 18.57s, 1 test, 1 error <<< FAILURES!

[...truncated 43623 lines...]
-ecj-javadoc-lint-src:
    [mkdir] Created dir: /var/tmp/ecj112807485
 [ecj-lint] Compiling 1149 source files to /var/tmp/ecj112807485
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. ERROR in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/cloud/AddReplicaCmd.java (at line 27)
 [ecj-lint] 	import java.util.concurrent.atomic.AtomicBoolean;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import java.util.concurrent.atomic.AtomicBoolean is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 2. ERROR in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/cloud/autoscaling/AutoScaling.java (at line 26)
 [ecj-lint] 	import com.google.common.base.Preconditions;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import com.google.common.base.Preconditions is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/CoreContainer.java (at line 1033)
 [ecj-lint] 	core = new SolrCore(this, dcore, coreConfig);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'core' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 4. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/core/HdfsDirectoryFactory.java (at line 234)
 [ecj-lint] 	dir = new BlockDirectory(path, hdfsDir, cache, null, blockCacheReadEnabled, false, cacheMerges, cacheReadOnce);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'dir' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 121)
 [ecj-lint] 	reader = cfiltfac.create(reader);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'reader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 6. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 145)
 [ecj-lint] 	return namedList;
 [ecj-lint] 	^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'listBasedTokenStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 7. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java (at line 1285)
 [ecj-lint] 	DirectoryReader reader = s==null ? null : s.get().getIndexReader();
 [ecj-lint] 	                ^^^^^^
 [ecj-lint] Resource leak: 'reader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 8. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 517)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zk, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] 9. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 743)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zkHost, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java (at line 578)
 [ecj-lint] 	tvWindowStream = new OffsetWindowTokenFilter(tvStream);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tvWindowStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/request/SimpleFacets.java (at line 925)
 [ecj-lint] 	fastForRandomSet = new HashDocSet(sset.getDocs(), 0, sset.size());
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fastForRandomSet' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java (at line 33)
 [ecj-lint] 	new SmileWriter(out, request, response).writeResponse();
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/schema/OpenExchangeRatesOrgProvider.java (at line 146)
 [ecj-lint] 	ratesJsonStream = resourceLoader.openResource(ratesFileLocation);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'ratesJsonStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java (at line 1315)
 [ecj-lint] 	UninvertingReader uninvertingReader = new UninvertingReader(new ReaderWrapper(searcher.getSlowAtomicReader(), collapseField), mapping);
 [ecj-lint] 	                  ^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'uninvertingReader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 145)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 16. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 154)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 17. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 163)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/OrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/search/function/ReverseOrdFieldSource.java (at line 79)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 20. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java (at line 291)
 [ecj-lint] 	throw new SolrException(ErrorCode.SERVER_ERROR, "Error occurred while loading solr.xml from zookeeper", e);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'loader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/CdcrTransactionLog.java (at line 98)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 191)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 261)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 24. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 25. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 228)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 26. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 260)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 27. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 275)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 28. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 529)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 29. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] 30. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 31. WARNING in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/java/org/apache/solr/util/SolrLogLayout.java (at line 149)
 [ecj-lint] 	SolrQueryRequest req = requestInfo == null ? null : requestInfo.getReq();
 [ecj-lint] 	                 ^^^
 [ecj-lint] Resource leak: 'req' is never closed
 [ecj-lint] ----------
 [ecj-lint] 31 problems (2 errors, 29 warnings)

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:689: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2081: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2120: Compile failed; see the compiler error output for details.

Total time: 93 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1452 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1452/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication

Error Message:
Index: 0, Size: 0

Stack Trace:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at __randomizedtesting.SeedInfo.seed([7C1E330DBD08839F:685668589E0F3E81]:0)
	at java.util.ArrayList.rangeCheck(ArrayList.java:653)
	at java.util.ArrayList.get(ArrayList.java:429)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication(TestReplicationHandler.java:561)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testExecutorStream

Error Message:
Error from server at http://127.0.0.1:33029/solr/workQueue_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/workQueue_shard2_replica_n2/update. Reason: <pre>    Can not find: /solr/workQueue_shard2_replica_n2/update</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://127.0.0.1:33029/solr/workQueue_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/workQueue_shard2_replica_n2/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([EAD91D9727B41A1:2C6D102251116BB1]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
	at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
	at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testExecutorStream(StreamExpressionTest.java:7471)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33029/solr/workQueue_shard2_replica_n2: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n2/update. Reason:
<pre>    Can not find: /solr/workQueue_shard2_replica_n2/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>

	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more




Build Log:
[...truncated 12564 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> 1856555 INFO  (SUITE-TestReplicationHandler-seed#[7C1E330DBD08839F]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/init-core-data-001
   [junit4]   2> 1856555 WARN  (SUITE-TestReplicationHandler-seed#[7C1E330DBD08839F]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=16 numCloses=16
   [junit4]   2> 1856556 INFO  (SUITE-TestReplicationHandler-seed#[7C1E330DBD08839F]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1856557 INFO  (SUITE-TestReplicationHandler-seed#[7C1E330DBD08839F]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1856558 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestStressReplication
   [junit4]   2> 1856558 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/collection1
   [junit4]   2> 1856561 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1856561 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@746b077f{/solr,null,AVAILABLE}
   [junit4]   2> 1856562 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2f68bc42{HTTP/1.1,[http/1.1]}{127.0.0.1:43091}
   [junit4]   2> 1856562 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server Started @1860025ms
   [junit4]   2> 1856562 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=43091}
   [junit4]   2> 1856563 ERROR (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856563 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1856563 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1856563 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1856563 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-03T22:13:27.794Z
   [junit4]   2> 1856563 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/solr.xml
   [junit4]   2> 1856566 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1856596 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856602 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856602 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856604 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/.
   [junit4]   2> 1856604 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1856604 INFO  (coreLoadExecutor-3880-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1856612 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1856625 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1856627 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1856627 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1856627 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856627 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1856627 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1856628 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=75.7265625, floorSegmentMB=0.65234375, forceMergeDeletesPctAllowed=11.494274922496768, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1856660 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1856660 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1856661 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=54.5, floorSegmentMB=1.8427734375, forceMergeDeletesPctAllowed=20.004781891310305, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1856661 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@688fe321[collection1] main]
   [junit4]   2> 1856661 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/collection1/conf
   [junit4]   2> 1856662 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1856662 INFO  (coreLoadExecutor-3880-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1856662 INFO  (searcherExecutor-3881-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@688fe321[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1856663 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1
   [junit4]   2> 1856665 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1856666 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2480f826{/solr,null,AVAILABLE}
   [junit4]   2> 1856667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2076d78a{HTTP/1.1,[http/1.1]}{127.0.0.1:39225}
   [junit4]   2> 1856667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server Started @1860130ms
   [junit4]   2> 1856667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=39225}
   [junit4]   2> 1856667 ERROR (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1856667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1856667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1856667 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-03T22:13:27.898Z
   [junit4]   2> 1856668 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/solr.xml
   [junit4]   2> 1856671 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1856698 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856705 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856705 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856706 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/.
   [junit4]   2> 1856706 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1856707 INFO  (coreLoadExecutor-3890-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1856716 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1856720 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1856721 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1856722 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1856722 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856722 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1856722 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1856732 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=75.7265625, floorSegmentMB=0.65234375, forceMergeDeletesPctAllowed=11.494274922496768, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1856769 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1856769 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1856769 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=54.5, floorSegmentMB=1.8427734375, forceMergeDeletesPctAllowed=20.004781891310305, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1856770 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@488ca840[collection1] main]
   [junit4]   2> 1856770 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1/conf
   [junit4]   2> 1856771 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1856771 INFO  (coreLoadExecutor-3890-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1856772 INFO  (searcherExecutor-3891-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@488ca840[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1856774 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@2076d78a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1856774 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1495668197
   [junit4]   2> 1856774 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1856774 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1d2cce2c: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@63e86dc9
   [junit4]   2> 1856779 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1856780 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6c49365a: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@1691e6c9
   [junit4]   2> 1856782 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1856782 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@60363d98: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1736b3c5
   [junit4]   2> 1856783 INFO  (coreCloseExecutor-3896-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@651450b8
   [junit4]   2> 1856784 INFO  (coreCloseExecutor-3896-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1695830200
   [junit4]   2> 1856784 INFO  (coreCloseExecutor-3896-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3aa8fe75: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@661840ee
   [junit4]   2> 1856791 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2480f826{/solr,null,UNAVAILABLE}
   [junit4]   2> 1856793 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1856794 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@327c1817{/solr,null,AVAILABLE}
   [junit4]   2> 1856794 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4389c975{HTTP/1.1,[http/1.1]}{127.0.0.1:48314}
   [junit4]   2> 1856794 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server Started @1860258ms
   [junit4]   2> 1856794 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=48314}
   [junit4]   2> 1856795 ERROR (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856795 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1856795 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1856795 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1856795 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-03T22:13:28.026Z
   [junit4]   2> 1856796 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/solr.xml
   [junit4]   2> 1856799 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1856829 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856833 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856833 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856833 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/.
   [junit4]   2> 1856833 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1856838 INFO  (coreLoadExecutor-3903-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1856847 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1856851 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1856852 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1856852 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1, trusted=true
   [junit4]   2> 1856853 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856853 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1856853 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1856854 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15127797993892625]
   [junit4]   2> 1856899 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1856899 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1856900 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.705818942854619]
   [junit4]   2> 1856900 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1dc1063[collection1] main]
   [junit4]   2> 1856900 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1/conf
   [junit4]   2> 1856901 INFO  (coreLoadExecutor-3903-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1856901 INFO  (searcherExecutor-3904-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1dc1063[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1856903 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@2f68bc42{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1856903 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1580894958
   [junit4]   2> 1856903 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
   [junit4]   2> 1856903 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6c61c044: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@65b76bdf
   [junit4]   2> 1856905 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
   [junit4]   2> 1856905 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6d0cd708: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@1691e6c9
   [junit4]   2> 1856907 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
   [junit4]   2> 1856907 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@21112a6e: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1736b3c5
   [junit4]   2> 1856908 INFO  (coreCloseExecutor-3908-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7042a832
   [junit4]   2> 1856909 INFO  (coreCloseExecutor-3908-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1883416626
   [junit4]   2> 1856909 INFO  (coreCloseExecutor-3908-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@ddb798f: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@344b0c5c
   [junit4]   2> 1856914 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@746b077f{/solr,null,UNAVAILABLE}
   [junit4]   2> 1856916 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server jetty-9.3.20.v20170531
   [junit4]   2> 1856917 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7ba1e889{/solr,null,AVAILABLE}
   [junit4]   2> 1856917 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@33ad50a6{HTTP/1.1,[http/1.1]}{127.0.0.1:45000}
   [junit4]   2> 1856917 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.e.j.s.Server Started @1860380ms
   [junit4]   2> 1856917 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=45000}
   [junit4]   2> 1856917 ERROR (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1856917 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.0.0
   [junit4]   2> 1856918 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1856918 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null, Default config dir: null
   [junit4]   2> 1856918 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-10-03T22:13:28.149Z
   [junit4]   2> 1856918 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/solr.xml
   [junit4]   2> 1856921 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 1856951 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856957 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856958 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856959 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/.
   [junit4]   2> 1856959 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1856960 INFO  (coreLoadExecutor-3915-thread-1) [    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 1856968 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1856973 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1856975 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1856975 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/./collection1, trusted=true
   [junit4]   2> 1856975 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1856975 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 1856975 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1856976 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15127797993892625]
   [junit4]   2> 1857023 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1857023 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1857024 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.705818942854619]
   [junit4]   2> 1857024 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@76acbec1[collection1] main]
   [junit4]   2> 1857024 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-001/collection1/conf
   [junit4]   2> 1857025 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1857025 INFO  (coreLoadExecutor-3915-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1857026 INFO  (searcherExecutor-3916-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@76acbec1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1857028 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1857029 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 0
   [junit4]   2> 1857031 INFO  (qtp420576524-12655) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1857031 INFO  (qtp420576524-12655) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 0
   [junit4]   2> 1857032 INFO  (qtp420576524-12656) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1857032 INFO  (qtp420576524-12656) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@3dd5cdd8 commitCommandVersion:0
   [junit4]   2> 1857035 INFO  (qtp420576524-12656) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@372da6a8[collection1] main]
   [junit4]   2> 1857036 INFO  (qtp420576524-12656) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1857036 INFO  (searcherExecutor-3904-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@372da6a8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1857037 INFO  (qtp420576524-12656) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 4
   [junit4]   2> 1857038 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 0
   [junit4]   2> 1857039 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 1857039 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 1857039 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 1857040 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 1857040 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 1857040 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 1857041 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 1857041 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 1857041 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 1857042 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 1857042 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 1857042 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 1857043 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 1857043 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 1857044 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 1857044 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 1857044 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 1857045 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 1857045 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 1857045 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 1857046 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 1857046 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 1857046 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 1857047 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 1857047 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 1857047 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 1857048 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 1857048 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 1857048 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 1857049 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 1857049 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 1857049 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 1857050 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 1857050 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 1857050 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 1857051 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 1857051 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 1857051 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 1857051 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 1857052 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 1857052 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 1857052 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 1857053 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 1857053 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 1857053 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 1857053 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 1857054 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 1857054 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 1857054 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 1857054 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 1857055 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 1857055 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 1857055 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 1857055 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 1857056 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 1857056 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 1857056 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 1857056 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 1857057 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 1857057 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 1857057 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 1857058 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 1857058 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 1857058 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 1857059 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 1857059 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 1857059 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 1857059 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 1857060 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 1857060 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 1857061 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 1857061 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 1857061 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 1857061 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 1857062 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 1857062 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 1857063 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 1857063 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 1857063 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 1857064 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 1857064 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 1857064 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 1857065 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 1857065 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 1857065 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 1857066 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 1857066 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 1857066 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 1857067 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 1857067 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 1857067 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 1857067 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 1857068 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 1857068 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 1857068 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 1857068 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 1857068 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 1857069 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 1857069 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 1857069 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 1857070 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 1857070 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 1857070 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 1857071 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 1857071 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 1857071 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 1857071 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 1857071 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 1857073 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 1857073 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 1857073 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 1857074 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 1857074 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 1857074 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 1857075 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 1857075 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 1857075 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 1857076 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 1857076 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 1857076 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 1857077 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 1857077 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 1857077 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 1857078 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 1857078 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 1857078 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 1857079 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 0
   [junit4]   2> 1857079 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 1857080 INFO  (qtp1138952998-12678) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 1857080 INFO  (qtp1138952998-12675) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 1857080 INFO  (qtp1138952998-12676) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 1857081 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1857081 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@29d4b274 commitCommandVersion:0
   [junit4]   2> 1857088 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6fec4eeb[collection1] main]
   [junit4]   2> 1857089 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1857089 INFO  (searcherExecutor-3916-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6fec4eeb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(8.0.0):c132)))}
   [junit4]   2> 1857090 INFO  (qtp1138952998-12672) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 8
   [junit4]   2> 1857094 INFO  (qtp1138952998-12677) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=132 status=0 QTime=3
   [junit4]   2> 1857094 INFO  (TEST-TestReplicationHandler.doTestStressReplication-seed#[7C1E330DBD08839F]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 132 docs
   [junit4]   2> 1857096 INFO  (qtp420576524-12658) [    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={core=collection1&action=status&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1857100 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1857101 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1857102 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 1857102 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1507068808312
   [junit4]   2> 1857102 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 2
   [junit4]   2> 1857102 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's version: 1507068808263
   [junit4]   2> 1857102 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1857104 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=2
   [junit4]   2> 1857104 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 4
   [junit4]   2> 1857105 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=true) to NRTCachingDirectory(MMapDirectory@/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1/data/index.20171003171328336 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2b3f050a; maxCacheMB=48.0 maxMergeSizeMB=4.0)
   [junit4]   2> 1857105 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.cfe&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1857107 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1857107 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.cfs&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1857108 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher File segments_2 did not match. expected checksum is 2177212637 and actual is checksum 2357558756. expected length is 185 and actual length is 120
   [junit4]   2> 1857109 INFO  (qtp1138952998-12671) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1857109 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=true,bytesDownloaded=5386) : 0 secs (null bytes/sec) to NRTCachingDirectory(MMapDirectory@/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1/data/index.20171003171328336 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2b3f050a; maxCacheMB=48.0 maxMergeSizeMB=4.0)
   [junit4]   2> 1857109 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download of configuration files from master: [{size=1530, name=schema.xml, checksum=1508380349}]
   [junit4]   2> 1857119 INFO  (qtp1138952998-12674) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&cf=schema.xml&checksum=true&wt=filestream&command=filecontent} status=0 QTime=4
   [junit4]   2> 1857122 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrCore Updating index properties... index=index.20171003171328336
   [junit4]   2> 1857123 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Configuration files are modified, core will be reloaded
   [junit4]   2> 1857124 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Reloading SolrCore collection1
   [junit4]   2> 1857132 INFO  (Thread-4499) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1857137 INFO  (Thread-4499) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1857138 INFO  (Thread-4499) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1857138 INFO  (Thread-4499) [    x:collection1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1
   [junit4]   2> 1857155 INFO  (Thread-4499) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c829ccc
   [junit4]   2> 1857155 INFO  (Thread-4499) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1857157 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.DirectoryFactory Will not remove most recent old directory after reload /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 1857157 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.DirectoryFactory Found 0 old index directories to clean-up under /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/./collection1/data/ afterReload=true
   [junit4]   2> 1857197 INFO  (Thread-4499) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1857197 INFO  (Thread-4499) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1857198 INFO  (Thread-4499) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@744e6c67[collection1] main]
   [junit4]   2> 1857198 INFO  (Thread-4499) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_7C1E330DBD08839F-001/solr-instance-002/collection1/conf
   [junit4]   2> 1857198 INFO  (Thread-4499) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1857199 INFO  (Thread-4499) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15127797993892625]
   [junit4]   2> 1857200 INFO  (searcherExecutor-3921-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@744e6c67[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1857200 INFO  (Thread-4499) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1857200 INFO  (Thread-4499) [    x:collection1] o.a.s.s.SolrIndexSearcher 

[...truncated too long message...]

41928-127.0.0.1:36452_solr-n_0000000000) closing
   [junit4]   2> 213896 INFO  (OverseerStateUpdate-98767455426641928-127.0.0.1:36452_solr-n_0000000000) [n:127.0.0.1:36452_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:36452_solr
   [junit4]   2> 213903 INFO  (zkCallback-337-thread-1-processing-n:127.0.0.1:50059_solr) [n:127.0.0.1:50059_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50059_solr
   [junit4]   2> 213906 INFO  (zkCallback-337-thread-3-processing-n:127.0.0.1:50059_solr) [n:127.0.0.1:50059_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 213912 INFO  (jetty-closer-316-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@16d8b2b8{/solr,null,UNAVAILABLE}
   [junit4]   2> 215358 WARN  (zkCallback-336-thread-4-processing-n:127.0.0.1:33029_solr) [n:127.0.0.1:33029_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 215358 INFO  (jetty-closer-316-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@30c50ffa{/solr,null,UNAVAILABLE}
   [junit4]   2> 215419 WARN  (zkCallback-337-thread-1-processing-n:127.0.0.1:50059_solr) [n:127.0.0.1:50059_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 215419 INFO  (jetty-closer-316-thread-4) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5a7bb3d5{/solr,null,UNAVAILABLE}
   [junit4]   2> 215421 ERROR (SUITE-StreamExpressionTest-seed#[EAD91D9727B41A1]-worker) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 215421 INFO  (SUITE-StreamExpressionTest-seed#[EAD91D9727B41A1]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39171 39171
   [junit4]   2> 225465 INFO  (Thread-219) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39171 39171
   [junit4]   2> 225467 WARN  (Thread-219) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	78	/solr/clusterprops.json
   [junit4]   2> 	78	/solr/aliases.json
   [junit4]   2> 	8	/solr/configs/conf
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	78	/solr/clusterstate.json
   [junit4]   2> 	22	/solr/collections/destination/state.json
   [junit4]   2> 	22	/solr/collections/mainCorpus/state.json
   [junit4]   2> 	20	/solr/collections/parallelDestinationCollection/state.json
   [junit4]   2> 	20	/solr/collections/workQueue/state.json
   [junit4]   2> 	20	/solr/collections/destinationCollection/state.json
   [junit4]   2> 	18	/solr/collections/parallelDestinationCollection1/state.json
   [junit4]   2> 	12	/solr/collections/collection1_collection/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/98767455426641931-127.0.0.1:33029_solr-n_0000000001
   [junit4]   2> 	2	/solr/overseer_elect/election/98767455426641928-127.0.0.1:36452_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	78	/solr/live_nodes
   [junit4]   2> 	78	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_EAD91D9727B41A1-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=PostingsFormat(name=Memory), join2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), subject=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), price_f=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), intDefault=FST50, a_f=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a2_s=Lucene50(blocksize=128), join1_i=PostingsFormat(name=Memory), a_i=PostingsFormat(name=Memory), test_i=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), join3_i=Lucene50(blocksize=128), field_i=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), level1_s=PostingsFormat(name=Memory), body_t=Lucene50(blocksize=128), s_multi=Lucene50(blocksize=128), col_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), level2_s=FST50, a_ss=FST50, a_s=Lucene50(blocksize=128), a_t=FST50, id=PostingsFormat(name=Memory), text=Lucene50(blocksize=128), timestamp=FST50, i_multi=PostingsFormat(name=Memory), b_ls=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), test_dt=PostingsFormat(name=Memory), test_t=Lucene50(blocksize=128), ident_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), order_i=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), _version_=FST50, side_s=Lucene50(blocksize=128), miles_i=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{expr_s=DocValuesFormat(name=Lucene70), join2_s=DocValuesFormat(name=Lucene70), multiDefault=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Lucene70), a1_s=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Memory), a_f=DocValuesFormat(name=Lucene70), a2_s=DocValuesFormat(name=Direct), join1_i=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Lucene70), test_i=DocValuesFormat(name=Lucene70), join3_i=DocValuesFormat(name=Direct), field_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Lucene70), s_multi=DocValuesFormat(name=Direct), col_s=DocValuesFormat(name=Lucene70), level2_s=DocValuesFormat(name=Memory), a_ss=DocValuesFormat(name=Memory), a_s=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Memory), i_multi=DocValuesFormat(name=Lucene70), b_ls=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Lucene70), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Memory), side_s=DocValuesFormat(name=Direct), miles_i=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=841, maxMBSortInHeap=7.019418034104843, sim=RandomSimilarity(queryNorm=true): {}, locale=de-AT, timezone=America/Creston
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=102801760,total=427646976
   [junit4]   2> NOTE: All tests run in this JVM: [JsonValidatorTest, TestClusteringResponse, TestSolrProperties, TestUpdateRequestCodec, HttpSolrClientConPoolTest, CollectionAdminRequestRequiredParamsTest, TangentEvaluatorTest, SolrDocumentTest, CoalesceEvaluatorTest, ArcTangentEvaluatorTest, TestSuggesterResponse, SolrExampleStreamingTest, AndEvaluatorTest, TestXMLEscaping, RoundEvaluatorTest, SolrPingTest, FieldValueEvaluatorTest, SolrSchemalessExampleTest, FieldAnalysisResponseTest, ConcurrentUpdateSolrClientTest, StreamExpressionParserTest, DivideEvaluatorTest, JdbcDriverTest, TestConfigSetAdminRequest, EqualToEvaluatorTest, ModifiableSolrParamsTest, TestCoreAdmin, SolrExampleXMLTest, TestPathTrie, CorrelationEvaluatorTest, OrEvaluatorTest, GraphExpressionTest, LBHttpSolrClientBuilderTest, JDBCStreamTest, HyperbolicCosineEvaluatorTest, JettyWebappTest, SolrExampleEmbeddedTest, TestBatchUpdate, LargeVolumeJettyTest, SolrParamTest, TermsResponseTest, TestEmbeddedSolrServer, TestJavaBinCodec, TestHash, TestFastInputStream, NamedListTest, FacetFieldTest, BasicHttpSolrClientTest, CloudSolrClientBuilderTest, CloudSolrClientCacheTest, CloudSolrClientTest, StreamExpressionTest]
   [junit4] Completed [137/145 (1!)] on J0 in 88.06s, 117 tests, 1 error, 9 skipped <<< FAILURES!

[...truncated 45540 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [exec] 
     [exec] file:///export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/docs/quickstart.html
     [exec]   BAD EXTERNAL LINK: https://lucene.apache.org/solr/guide/solr-tutorial.html
     [exec] 
     [exec] Broken javadocs links were found! Common root causes:
     [exec] * A typo of some sort for manually created links.
     [exec] * Public methods referencing non-public classes in their signature.

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:669: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:682: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2570: exec returned: 1

Total time: 110 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1451 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1451/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 60276 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [exec] 
     [exec] file:///export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/docs/quickstart.html
     [exec]   BAD EXTERNAL LINK: https://lucene.apache.org/solr/guide/solr-tutorial.html
     [exec] 
     [exec] Broken javadocs links were found! Common root causes:
     [exec] * A typo of some sort for manually created links.
     [exec] * Public methods referencing non-public classes in their signature.

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:669: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:682: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2570: exec returned: 1

Total time: 100 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1450 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1450/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 60306 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [exec] 
     [exec] file:///export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/docs/quickstart.html
     [exec]   BAD EXTERNAL LINK: https://lucene.apache.org/solr/guide/solr-tutorial.html
     [exec] 
     [exec] Broken javadocs links were found! Common root causes:
     [exec] * A typo of some sort for manually created links.
     [exec] * Public methods referencing non-public classes in their signature.

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:669: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:682: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2570: exec returned: 1

Total time: 96 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any