You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/01/22 02:59:38 UTC

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1089 - Still Failing!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1089/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
3 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    2) Thread[id=24576, name=jetty-launcher-4263-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)    3) Thread[id=24666, name=jetty-launcher-4263-thread-1-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   2) Thread[id=24576, name=jetty-launcher-4263-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   3) Thread[id=24666, name=jetty-launcher-4263-thread-1-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
	at __randomizedtesting.SeedInfo.seed([4F90935B06C1D593]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
	at __randomizedtesting.SeedInfo.seed([4F90935B06C1D593]:0)




Build Log:
[...truncated 12395 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_4F90935B06C1D593-001/init-core-data-001
   [junit4]   2> 2884106 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[4F90935B06C1D593]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2884108 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[4F90935B06C1D593]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2884145 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[4F90935B06C1D593]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_4F90935B06C1D593-001/tempDir-001
   [junit4]   2> 2884146 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[4F90935B06C1D593]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2884146 INFO  (Thread-5959) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2884146 INFO  (Thread-5959) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2884246 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[4F90935B06C1D593]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:35499
   [junit4]   2> 2884254 INFO  (jetty-launcher-4263-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2884254 INFO  (jetty-launcher-4263-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2884256 INFO  (jetty-launcher-4263-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@367a4890{/solr,null,AVAILABLE}
   [junit4]   2> 2884256 INFO  (jetty-launcher-4263-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6d00fb44{HTTP/1.1,[http/1.1]}{127.0.0.1:63538}
   [junit4]   2> 2884256 INFO  (jetty-launcher-4263-thread-1) [    ] o.e.j.s.Server Started @2889071ms
   [junit4]   2> 2884256 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=63538}
   [junit4]   2> 2884256 ERROR (jetty-launcher-4263-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@481ea680{/solr,null,AVAILABLE}
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-22T02:43:42.362Z
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@464f1d62{HTTP/1.1,[http/1.1]}{127.0.0.1:32997}
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.e.j.s.Server Started @2889072ms
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=32997}
   [junit4]   2> 2884257 ERROR (jetty-launcher-4263-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2884257 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-22T02:43:42.362Z
   [junit4]   2> 2884260 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2884260 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2884281 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35499/solr
   [junit4]   2> 2884281 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35499/solr
   [junit4]   2> 2884319 INFO  (jetty-launcher-4263-thread-2) [n:127.0.0.1:32997_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:32997_solr
   [junit4]   2> 2884320 INFO  (jetty-launcher-4263-thread-1) [n:127.0.0.1:63538_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63538_solr
   [junit4]   2> 2884321 INFO  (jetty-launcher-4263-thread-2) [n:127.0.0.1:32997_solr    ] o.a.s.c.Overseer Overseer (id=97324434866307077-127.0.0.1:32997_solr-n_0000000000) starting
   [junit4]   2> 2884323 INFO  (zkCallback-4274-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2884323 INFO  (zkCallback-4273-thread-1-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2884335 INFO  (jetty-launcher-4263-thread-2) [n:127.0.0.1:32997_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32997_solr
   [junit4]   2> 2884337 INFO  (zkCallback-4274-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2884337 INFO  (zkCallback-4273-thread-1-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2884388 INFO  (jetty-launcher-4263-thread-1) [n:127.0.0.1:63538_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 2884391 INFO  (jetty-launcher-4263-thread-1) [n:127.0.0.1:63538_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=solaris-vm,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 2884396 INFO  (jetty-launcher-4263-thread-1) [n:127.0.0.1:63538_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2884403 INFO  (jetty-launcher-4263-thread-1-EventThread) [n:127.0.0.1:63538_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2884433 INFO  (jetty-launcher-4263-thread-1) [n:127.0.0.1:63538_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_4F90935B06C1D593-001/tempDir-001/node1/.
   [junit4]   2> 2884582 INFO  (jetty-launcher-4263-thread-2) [n:127.0.0.1:32997_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 2884583 INFO  (jetty-launcher-4263-thread-2) [n:127.0.0.1:32997_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=solaris-vm,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 2884583 INFO  (jetty-launcher-4263-thread-2) [n:127.0.0.1:32997_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2884586 INFO  (jetty-launcher-4263-thread-2-EventThread) [n:127.0.0.1:32997_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2884610 INFO  (jetty-launcher-4263-thread-2) [n:127.0.0.1:32997_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_4F90935B06C1D593-001/tempDir-001/node2/.
   [junit4]   2> 2885292 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 2885327 WARN  (qtp1858694917-24521) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
   [junit4]   2> 
   [junit4]   2> 2885327 WARN  (qtp1858694917-24521) [    ] o.a.h.s.UserGroupInformation No groups available for user bar
   [junit4]   2> 2885331 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 2885360 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testForwarding
   [junit4]   2> 2886044 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[4F90935B06C1D593]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2886046 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[4F90935B06C1D593]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35499/solr ready
   [junit4]   2> 2886048 INFO  (qtp1858694917-24523) [n:127.0.0.1:63538_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2886053 INFO  (OverseerThreadFactory-8882-thread-1-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.CreateCollectionCmd Create collection forwardingCollection
   [junit4]   2> 2886162 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 2886270 INFO  (zkCallback-4274-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887187 WARN  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 2887188 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2887211 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.s.IndexSchema [forwardingCollection_shard1_replica1] Schema name=test
   [junit4]   2> 2887307 WARN  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.s.IndexSchema [forwardingCollection_shard1_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2887311 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2887331 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'forwardingCollection_shard1_replica1' using configuration from collection forwardingCollection
   [junit4]   2> 2887332 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.SolrCore [[forwardingCollection_shard1_replica1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_4F90935B06C1D593-001/tempDir-001/node1/forwardingCollection_shard1_replica1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_4F90935B06C1D593-001/tempDir-001/node1/./forwardingCollection_shard1_replica1/data/]
   [junit4]   2> 2887332 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4dc0a2b6
   [junit4]   2> 2887335 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2887369 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2887370 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2887371 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2887371 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2887372 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=44, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6564531016588597]
   [junit4]   2> 2887373 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3b2c7438[forwardingCollection_shard1_replica1] main]
   [junit4]   2> 2887375 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 2887380 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2887398 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2887398 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 2887400 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 2887400 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 2887400 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557190961271930880
   [junit4]   2> 2887400 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 2887401 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 2887401 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 2887402 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 2887403 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 2887403 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 2887404 INFO  (searcherExecutor-8887-thread-1-processing-n:127.0.0.1:63538_solr x:forwardingCollection_shard1_replica1 s:shard1 c:forwardingCollection) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica1] Registered new searcher Searcher@3b2c7438[forwardingCollection_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2887411 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2887412 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2887412 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:63538/solr/forwardingCollection_shard1_replica1/
   [junit4]   2> 2887412 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2887412 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.SyncStrategy http://127.0.0.1:63538/solr/forwardingCollection_shard1_replica1/ has no replicas
   [junit4]   2> 2887423 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:63538/solr/forwardingCollection_shard1_replica1/ shard1
   [junit4]   2> 2887527 INFO  (zkCallback-4274-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2887575 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2887579 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2} status=0 QTime=1417
   [junit4]   2> 2887584 INFO  (qtp1858694917-24523) [n:127.0.0.1:63538_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 2887682 INFO  (zkCallback-4274-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2888057 INFO  (OverseerCollectionConfigSetProcessor-97324434866307077-127.0.0.1:32997_solr-n_0000000000) [n:127.0.0.1:32997_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 2888584 INFO  (qtp1858694917-24523) [n:127.0.0.1:63538_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=2536
   [junit4]   2> 2888585 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[4F90935B06C1D593]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: forwardingCollection failOnTimeout:true timeout (sec):100
   [junit4]   2> 2888585 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[4F90935B06C1D593]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: forwardingCollection
   [junit4]   2> 2888590 INFO  (qtp1858694917-24519) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.S.Request [forwardingCollection_shard1_replica1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2888597 INFO  (qtp1858694917-24524) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.S.Request [forwardingCollection_shard1_replica1]  webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2888597 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testForwarding
   [junit4]   2> 2888630 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 2888632 INFO  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:128)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:404)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:310)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2888633 WARN  (qtp1858694917-24525) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1  x:forwardingCollection_shard1_replica1] o.e.j.s.ServletHandler /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication, 
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:410)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:310)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:128)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:404)
   [junit4]   2> 	... 26 more
   [junit4]   2> 2888634 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4]   2> 2888662 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidRemote
   [junit4]   2> 2898675 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidRemote
   [junit4]   2> 2898716 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
   [junit4]   2> 2919030 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
   [junit4]   2> 2919063 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 2919064 INFO  (qtp1858694917-24521) [n:127.0.0.1:63538_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2919064 INFO  (qtp1858694917-24521) [n:127.0.0.1:63538_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2919064 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4]   2> 2919095 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 2919128 WARN  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@58b5b318 name: ZooKeeperConnection Watcher:127.0.0.1:35499/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2919128 WARN  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2919129 WARN  (zkCallback-4273-thread-4-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1366302a name: ZooKeeperConnection Watcher:127.0.0.1:35499/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2919129 WARN  (zkCallback-4273-thread-4-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2919129 INFO  (jetty-launcher-4263-thread-1-EventThread) [n:127.0.0.1:63538_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 2919135 WARN  (zkCallback-4274-thread-2-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@12c05b9b name: ZooKeeperConnection Watcher:127.0.0.1:35499/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2919135 WARN  (zkCallback-4274-thread-2-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2924110 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 2924112 WARN  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@12c05b9b name: ZooKeeperConnection Watcher:127.0.0.1:35499/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2924117 WARN  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@58b5b318 name: ZooKeeperConnection Watcher:127.0.0.1:35499/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2924119 WARN  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2924122 WARN  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2924115 WARN  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1366302a name: ZooKeeperConnection Watcher:127.0.0.1:35499/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2924122 WARN  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2924122 WARN  (OverseerStateUpdate-97324434866307077-127.0.0.1:32997_solr-n_0000000000) [n:127.0.0.1:32997_solr    ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:339)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:336)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:384)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:160)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:138)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:179)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2924123 INFO  (OverseerStateUpdate-97324434866307077-127.0.0.1:32997_solr-n_0000000000) [n:127.0.0.1:32997_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:32997_solr
   [junit4]   2> 2924119 WARN  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2924125 INFO  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2924125 ERROR (OverseerExitThread) [n:127.0.0.1:32997_solr    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:286)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2924127 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.Overseer Overseer (id=97324434866307077-127.0.0.1:32997_solr-n_0000000000) closing
   [junit4]   2> 2924127 WARN  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2924128 WARN  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2924129 INFO  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 2924132 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2924132 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2924133 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2924133 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2924136 INFO  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 2924136 INFO  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2924137 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 2924137 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 2924168 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:32997_solr
   [junit4]   2> 2924175 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 2924176 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.Overseer Overseer (id=97324434866307085-127.0.0.1:32997_solr-n_0000000002) starting
   [junit4]   2> 2924177 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 2924188 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63538_solr
   [junit4]   2> 2924209 INFO  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2924209 INFO  (zkCallback-4273-thread-20-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2924214 INFO  (zkCallback-4274-thread-4-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2924215 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 2924215 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2924215 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.ZkController Registering core forwardingCollection_shard1_replica1 afterExpiration? true
   [junit4]   2> 2924217 INFO  (Thread-5983) [n:127.0.0.1:63538_solr    ] o.a.s.c.SolrCore config update listener called for core forwardingCollection_shard1_replica1
   [junit4]   2> 2924220 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32997_solr
   [junit4]   2> 2924225 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.ActionThrottle The last leader attempt started 36823ms ago.
   [junit4]   2> 2924233 INFO  (zkCallback-4281-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2924239 INFO  (zkCallback-4273-thread-20-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2924239 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
   [junit4]   2> 2924239 INFO  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2924247 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2924250 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2924251 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2924251 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:63538/solr/forwardingCollection_shard1_replica1/
   [junit4]   2> 2924251 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2924251 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.SyncStrategy http://127.0.0.1:63538/solr/forwardingCollection_shard1_replica1/ has no replicas
   [junit4]   2> 2924257 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2924258 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 2924260 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 2924261 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:63538/solr/forwardingCollection_shard1_replica1/ shard1
   [junit4]   2> 2924270 INFO  (coreZkRegister-8873-thread-1-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2924330 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 2924366 INFO  (qtp1858694917-24523) [n:127.0.0.1:63538_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2924366 INFO  (qtp1858694917-24523) [n:127.0.0.1:63538_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2924366 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4]   2> 2924407 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 2924410 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 2924460 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 2924461 INFO  (qtp1858694917-24522) [n:127.0.0.1:63538_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2924462 INFO  (qtp1858694917-24522) [n:127.0.0.1:63538_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2924462 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[4F90935B06C1D593]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 2924469 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 2924469 INFO  (jetty-launcher-4263-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6d00fb44{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2924470 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1792239713
   [junit4]   2> 2924471 INFO  (jetty-launcher-4263-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@464f1d62{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2924472 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1121397160
   [junit4]   2> 2924474 INFO  (coreCloseExecutor-8894-thread-1) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3fd1e932
   [junit4]   2> 2924478 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.c.Overseer Overseer (id=97324434866307085-127.0.0.1:32997_solr-n_0000000002) closing
   [junit4]   2> 2924479 INFO  (OverseerStateUpdate-97324434866307085-127.0.0.1:32997_solr-n_0000000002) [n:127.0.0.1:32997_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:32997_solr
   [junit4]   2> 2924483 WARN  (zkCallback-4273-thread-5-processing-n:127.0.0.1:32997_solr) [n:127.0.0.1:32997_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2924484 INFO  (jetty-launcher-4263-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2924487 ERROR (Thread[Thread-5979,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:32997_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2924493 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2924495 INFO  (jetty-launcher-4263-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@481ea680{/solr,null,UNAVAILABLE}
   [junit4]   2> 2924496 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:63538_solr
   [junit4]   2> 2924501 INFO  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.Overseer Overseer (id=97324434866307086-127.0.0.1:63538_solr-n_0000000003) starting
   [junit4]   2> 2924507 INFO  (coreCloseExecutor-8894-thread-1) [n:127.0.0.1:63538_solr c:forwardingCollection s:shard1 r:core_node1 x:forwardingCollection_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.forwardingCollection.shard1.replica1
   [junit4]   2> 2924511 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.c.Overseer Overseer (id=97324434866307086-127.0.0.1:63538_solr-n_0000000003) closing
   [junit4]   2> 2924511 INFO  (OverseerCollectionConfigSetProcessor-97324434866307086-127.0.0.1:63538_solr-n_0000000003) [n:127.0.0.1:63538_solr    ] o.a.s.c.OverseerTaskProcessor According to ZK I (id=97324434866307086-127.0.0.1:63538_solr-n_0000000003) am no longer a leader.
   [junit4]   2> 2924512 INFO  (OverseerStateUpdate-97324434866307086-127.0.0.1:63538_solr-n_0000000003) [n:127.0.0.1:63538_solr    ] o.a.s.c.Overseer According to ZK I (id=97324434866307086-127.0.0.1:63538_solr-n_0000000003) am no longer a leader.
   [junit4]   2> 2924512 INFO  (OverseerStateUpdate-97324434866307086-127.0.0.1:63538_solr-n_0000000003) [n:127.0.0.1:63538_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:63538_solr
   [junit4]   2> 2924512 WARN  (zkCallback-4274-thread-6-processing-n:127.0.0.1:63538_solr) [n:127.0.0.1:63538_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2924514 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x159c40ed02c000e, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2924514 INFO  (jetty-launcher-4263-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2924515 ERROR (Thread[Thread-5976,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:63538_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2924519 INFO  (jetty-launcher-4263-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@367a4890{/solr,null,UNAVAILABLE}
   [junit4]   2> 2924522 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[4F90935B06C1D593]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35499 35499
   [junit4]   2> 2929545 INFO  (Thread-5959) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35499 35499
   [junit4]   2> 2929547 WARN  (Thread-5959) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 	2	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 	2	/solr/collections/forwardingCollection/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/overseer/collection-queue-work
   [junit4]   2> 	8	/solr/overseer/queue
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 	4	/solr/live_nodes
   [junit4]   2> 	3	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 2929581 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[4F90935B06C1D593]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Jan 22, 2017 2:44:27 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 2949190 ERROR (jetty-launcher-4263-thread-1-EventThread) [n:127.0.0.1:63538_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.IllegalStateException: Client is not started
   [junit4]   2> 	at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
   [junit4]   2> 	at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:477)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceFacade.getZooKeeper(NamespaceFacade.java:112)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 2949192 WARN  (jetty-launcher-4263-thread-1-EventThread) [n:127.0.0.1:63538_solr    ] o.a.c.ConnectionState Session expired event received
   [junit4]   2> 2949194 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> Jan 22, 2017 2:44:47 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 3 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]   2>    1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2>    2) Thread[id=24576, name=jetty-launcher-4263-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2>    3) Thread[id=24666, name=jetty-launcher-4263-thread-1-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]   2> Jan 22, 2017 2:44:47 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>    2) Thread[id=24576, name=jetty-launcher-4263-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>    3) Thread[id=24666, name=jetty-launcher-4263-thread-1-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2> 2949808 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.c.StaticHostProvider Unexpected exception
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 2949808 ERROR (jetty-launcher-4263-thread-1-EventThread) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Event thread exiting due to interruption
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]   2> 2949809 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2949810 ERROR (jetty-launcher-4263-thread-2-EventThread) [n:127.0.0.1:32997_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 2949810 WARN  (jetty-launcher-4263-thread-2-EventThread) [n:127.0.0.1:32997_solr    ] o.a.c.ConnectionState Session expired event received
   [junit4]   2> 2949811 WARN  (jetty-launcher-4263-thread-2-SendThread(127.0.0.1:35499)) [n:127.0.0.1:32997_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2950312 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.c.StaticHostProvider Unexpected exception
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 2950313 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2950820 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.c.StaticHostProvider Unexpected exception
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 2950828 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2950920 WARN  (jetty-launcher-4263-thread-2-SendThread(127.0.0.1:35499)) [n:127.0.0.1:32997_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2951320 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.c.StaticHostProvider Unexpected exception
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 2951321 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2951821 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.c.StaticHostProvider Unexpected exception
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 2951822 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2952023 WARN  (jetty-launcher-4263-thread-2-SendThread(127.0.0.1:35499)) [n:127.0.0.1:32997_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 2952322 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.c.StaticHostProvider Unexpected exception
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 2952323 WARN  (jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499)) [n:127.0.0.1:63538_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> Jan 22, 2017 2:44:50 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [ju

[...truncated too long message...]

rtitioner, TestReload, TestStressVersions, TestRangeQuery, SuggesterTest, SpatialFilterTest, SuggesterWFSTTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, DistanceFunctionTest, XsltUpdateRequestHandlerTest, CacheHeaderTest, LukeRequestHandlerTest, TestWriterPerf, PathHierarchyTokenizerFactoryTest, FieldAnalysisRequestHandlerTest, LoggingHandlerTest, SolrPluginUtilsTest, TestCSVResponseWriter, QueryParsingTest, TestPartialUpdateDeduplication, TestComponentsName, HighlighterConfigTest, UpdateParamsTest, AlternateDirectoryTest, ResponseLogComponentTest, SolrIndexConfigTest, TestStressRecovery, TestConfig, TestPostingsSolrHighlighter, TestPhraseSuggestions, SynonymTokenizerTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestJmxMonitoredMap, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest, DistributedIntervalFacetingTest, ActionThrottleTest, AsyncCallRequestStatusResponseTest, CollectionsAPISolrJTest, DistribDocExpirationUpdateProcessorTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, DocValuesNotIndexedTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, SolrXmlInZkTest, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestClusterProperties, TestCollectionAPI, TestConfigSetsAPI, TestConfigSetsAPIExclusivity, TestDeleteCollectionOnDownNodes, TestLeaderInitiatedRecoveryThread, TestLockTree, TestMiniSolrCloudCluster, TestSSLRandomization, TestSegmentSorting, TestSolrCloudWithSecureImpersonation]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=4F90935B06C1D593 -Dtests.slow=true -Dtests.locale=fr -Dtests.timezone=Pacific/Easter -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestSolrCloudWithSecureImpersonation (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]    >    1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    2) Thread[id=24576, name=jetty-launcher-4263-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]    >    3) Thread[id=24666, name=jetty-launcher-4263-thread-1-EventThread, state=WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4F90935B06C1D593]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=24665, name=jetty-launcher-4263-thread-1-SendThread(127.0.0.1:35499), state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4F90935B06C1D593]:0)
   [junit4] Completed [533/680 (1!)] on J0 in 68.74s, 12 tests, 2 errors <<< FAILURES!

[...truncated 43099 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene...
  [javadoc] Loading source files for package org.apache.lucene.analysis...
  [javadoc] Loading source files for package org.apache.lucene.analysis.standard...
  [javadoc] Loading source files for package org.apache.lucene.analysis.tokenattributes...
  [javadoc] Loading source files for package org.apache.lucene.codecs...
  [javadoc] Loading source files for package org.apache.lucene.codecs.blocktree...
  [javadoc] Loading source files for package org.apache.lucene.codecs.compressing...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene50...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene60...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene62...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene70...
  [javadoc] Loading source files for package org.apache.lucene.codecs.perfield...
  [javadoc] Loading source files for package org.apache.lucene.document...
  [javadoc] Loading source files for package org.apache.lucene.geo...
  [javadoc] Loading source files for package org.apache.lucene.index...
  [javadoc] Loading source files for package org.apache.lucene.search...
  [javadoc] Loading source files for package org.apache.lucene.search.similarities...
  [javadoc] Loading source files for package org.apache.lucene.search.spans...
  [javadoc] Loading source files for package org.apache.lucene.store...
  [javadoc] Loading source files for package org.apache.lucene.util...
  [javadoc] Loading source files for package org.apache.lucene.util.automaton...
  [javadoc] Loading source files for package org.apache.lucene.util.bkd...
  [javadoc] Loading source files for package org.apache.lucene.util.fst...
  [javadoc] Loading source files for package org.apache.lucene.util.graph...
  [javadoc] Loading source files for package org.apache.lucene.util.mutable...
  [javadoc] Loading source files for package org.apache.lucene.util.packed...
  [javadoc] Constructing Javadoc information...
  [javadoc] javadoc: error - Argument for -bottom contains JavaScript.
  [javadoc] Use --allow-script-in-comments to allow use of JavaScript.
  [javadoc] 1 error

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:775: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build.xml:138: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:746: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/core/build.xml:54: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2101: Javadoc returned 1

Total time: 83 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1098 - Still unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1098/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
	at __randomizedtesting.SeedInfo.seed([D7CE5BBDD31F896E:5F9A64677DE3E496]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
	at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
	at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
	at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([D7CE5BBDD31F896E:5CE9886C921922EA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:187)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
	at org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey(ShardSplitTest.java:436)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
	at __randomizedtesting.SeedInfo.seed([D7CE5BBDD31F896E:BF716E9703859B82]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
	at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10941 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithDelegationTokens
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithDelegationTokens_D7CE5BBDD31F896E-001/init-core-data-001
   [junit4]   2> 322384 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 322386 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 322388 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithDelegationTokens_D7CE5BBDD31F896E-001/tempDir-001
   [junit4]   2> 322388 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 322389 INFO  (Thread-1021) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 322389 INFO  (Thread-1021) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 322489 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:63500
   [junit4]   2> 322502 INFO  (jetty-launcher-307-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 322502 INFO  (jetty-launcher-307-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 322505 INFO  (jetty-launcher-307-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3f3a2ffa{/solr,null,AVAILABLE}
   [junit4]   2> 322510 INFO  (jetty-launcher-307-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6f2680d9{HTTP/1.1,[http/1.1]}{127.0.0.1:45993}
   [junit4]   2> 322510 INFO  (jetty-launcher-307-thread-2) [    ] o.e.j.s.Server Started @325338ms
   [junit4]   2> 322510 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45993}
   [junit4]   2> 322510 ERROR (jetty-launcher-307-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 322510 INFO  (jetty-launcher-307-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72a039b1{/solr,null,AVAILABLE}
   [junit4]   2> 322510 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 322510 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 322515 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 322519 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-26T03:01:00.387Z
   [junit4]   2> 322519 INFO  (jetty-launcher-307-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5bd8d11a{HTTP/1.1,[http/1.1]}{127.0.0.1:61525}
   [junit4]   2> 322519 INFO  (jetty-launcher-307-thread-1) [    ] o.e.j.s.Server Started @325347ms
   [junit4]   2> 322519 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61525}
   [junit4]   2> 322519 ERROR (jetty-launcher-307-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 322520 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 322520 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 322520 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 322520 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-26T03:01:00.388Z
   [junit4]   2> 322529 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 322530 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 322543 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63500/solr
   [junit4]   2> 322543 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63500/solr
   [junit4]   2> 322602 INFO  (jetty-launcher-307-thread-2) [n:127.0.0.1:45993_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45993_solr
   [junit4]   2> 322604 INFO  (jetty-launcher-307-thread-1) [n:127.0.0.1:61525_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61525_solr
   [junit4]   2> 322605 INFO  (jetty-launcher-307-thread-2) [n:127.0.0.1:45993_solr    ] o.a.s.c.Overseer Overseer (id=97347152134733829-127.0.0.1:45993_solr-n_0000000000) starting
   [junit4]   2> 322608 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:45993_solr) [n:127.0.0.1:45993_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 322608 INFO  (zkCallback-317-thread-1-processing-n:127.0.0.1:61525_solr) [n:127.0.0.1:61525_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 322626 INFO  (jetty-launcher-307-thread-2) [n:127.0.0.1:45993_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45993_solr
   [junit4]   2> 322628 INFO  (zkCallback-318-thread-1-processing-n:127.0.0.1:45993_solr) [n:127.0.0.1:45993_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 322628 INFO  (zkCallback-317-thread-1-processing-n:127.0.0.1:61525_solr) [n:127.0.0.1:61525_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 322764 INFO  (jetty-launcher-307-thread-2) [n:127.0.0.1:45993_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 322773 INFO  (jetty-launcher-307-thread-2) [n:127.0.0.1:45993_solr    ] o.a.s.s.KerberosPlugin Params: {token.valid=30, delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, token.validity=36000, signer.secret.provider.zookeeper.path=/token, type=kerberos, zk-dt-secret-manager.enable=true, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper}
   [junit4]   2> 322779 INFO  (jetty-launcher-307-thread-2) [n:127.0.0.1:45993_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 322781 INFO  (jetty-launcher-307-thread-2-EventThread) [n:127.0.0.1:45993_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 322798 INFO  (jetty-launcher-307-thread-1) [n:127.0.0.1:61525_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 322800 INFO  (jetty-launcher-307-thread-1) [n:127.0.0.1:61525_solr    ] o.a.s.s.KerberosPlugin Params: {token.valid=30, delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, token.validity=36000, signer.secret.provider.zookeeper.path=/token, type=kerberos, zk-dt-secret-manager.enable=true, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper}
   [junit4]   2> 322800 INFO  (jetty-launcher-307-thread-1) [n:127.0.0.1:61525_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 322804 INFO  (jetty-launcher-307-thread-1-EventThread) [n:127.0.0.1:61525_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 322837 INFO  (jetty-launcher-307-thread-1) [n:127.0.0.1:61525_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithDelegationTokens_D7CE5BBDD31F896E-001/tempDir-001/node1/.
   [junit4]   2> 323813 INFO  (jetty-launcher-307-thread-2) [n:127.0.0.1:45993_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithDelegationTokens_D7CE5BBDD31F896E-001/tempDir-001/node2/.
   [junit4]   2> 323926 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClient-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenSolrClient
   [junit4]   2> 323990 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClient-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324012 INFO  (qtp2044322013-2572) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuBz5igFZ_MSg-QECFG2PEdRs5mCQf7A6EJEGEUgen1XmB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324012 INFO  (qtp2044322013-2572) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuBz5igFZ_MSg-QECFG2PEdRs5mCQf7A6EJEGEUgen1XmB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 324014 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClient-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenSolrClient
   [junit4]   2> 324044 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenew
   [junit4]   2> 324091 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324092 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324100 INFO  (qtp2044322013-2571) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB1ZigFZ_MShWQICFKXKluOR43ZeFwCQEkTfXnRnhLLhB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324100 INFO  (qtp2044322013-2571) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB1ZigFZ_MShWQICFKXKluOR43ZeFwCQEkTfXnRnhLLhB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324105 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324118 INFO  (qtp1874391176-2560) [n:127.0.0.1:45993_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB1ZigFZ_MShWQICFKXKluOR43ZeFwCQEkTfXnRnhLLhB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324118 INFO  (qtp1874391176-2560) [n:127.0.0.1:45993_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB1ZigFZ_MShWQICFKXKluOR43ZeFwCQEkTfXnRnhLLhB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324133 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324134 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324136 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324157 INFO  (qtp2044322013-2574) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB22igFZ_MShtgMCFAR0VnryyAWLshlPmvy8aCpu75t8B3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324157 INFO  (qtp2044322013-2574) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB22igFZ_MShtgMCFAR0VnryyAWLshlPmvy8aCpu75t8B3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324165 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324166 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324169 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324177 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324178 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324181 INFO  (qtp2044322013-2574) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB22igFZ_MShtgMCFAR0VnryyAWLshlPmvy8aCpu75t8B3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324181 INFO  (qtp2044322013-2574) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB22igFZ_MShtgMCFAR0VnryyAWLshlPmvy8aCpu75t8B3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324200 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324202 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324205 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324250 INFO  (qtp2044322013-2570) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB38igFZ_MSh_AQCFLAAJIj8zznmYEKdsaNgzhGbG2LNB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324250 INFO  (qtp2044322013-2570) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB38igFZ_MSh_AQCFLAAJIj8zznmYEKdsaNgzhGbG2LNB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324266 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324267 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324270 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324278 INFO  (qtp1874391176-2564) [n:127.0.0.1:45993_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB38igFZ_MSh_AQCFLAAJIj8zznmYEKdsaNgzhGbG2LNB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324278 INFO  (qtp1874391176-2564) [n:127.0.0.1:45993_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB38igFZ_MSh_AQCFLAAJIj8zznmYEKdsaNgzhGbG2LNB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324284 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324286 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324290 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324327 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324328 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324329 INFO  (qtp2044322013-2571) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB5RigFZ_MSiUQUCFD2YjwgchcwtKfBvNytXlGv4SlMJB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324329 INFO  (qtp2044322013-2571) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB5RigFZ_MSiUQUCFD2YjwgchcwtKfBvNytXlGv4SlMJB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324339 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324341 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324342 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324353 INFO  (qtp1874391176-2559) [n:127.0.0.1:45993_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB5RigFZ_MSiUQUCFD2YjwgchcwtKfBvNytXlGv4SlMJB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324353 INFO  (qtp1874391176-2559) [n:127.0.0.1:45993_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB5RigFZ_MSiUQUCFD2YjwgchcwtKfBvNytXlGv4SlMJB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324363 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324365 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324371 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324377 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenew
   [junit4]   2> 324428 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancel
   [junit4]   2> 324465 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324467 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324470 ERROR (qtp2044322013-2572) [n:127.0.0.1:61525_solr    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_6]
   [junit4]   2> 324479 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324486 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324488 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324490 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324509 ERROR (qtp1874391176-2559) [n:127.0.0.1:45993_solr    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_7]
   [junit4]   2> 324519 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324520 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324522 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324530 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324532 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324534 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324536 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancel
   [junit4]   2> 324570 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancelFail
   [junit4]   2> 324586 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancelFail
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithDelegationTokens -Dtests.method=testDelegationTokenCancelFail -Dtests.seed=D7CE5BBDD31F896E -Dtests.slow=true -Dtests.locale=es-SV -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.05s J0 | TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<200> but was:<404>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D7CE5BBDD31F896E:BF716E9703859B82]:0)
   [junit4]    > 	at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
   [junit4]    > 	at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 324629 INFO  (TEST-TestSolrCloudWithDelegationTokens.testZNodePaths-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZNodePaths
   [junit4]   2> 324641 INFO  (TEST-TestSolrCloudWithDelegationTokens.testZNodePaths-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZNodePaths
   [junit4]   2> 324675 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenVerify
   [junit4]   2> 324683 INFO  (qtp2044322013-2568) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB_QigFZ_MSj0AoCFD0jyNev1oJIqnxb3_uCt_DNJ15kB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324683 INFO  (qtp2044322013-2568) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB_QigFZ_MSj0AoCFD0jyNev1oJIqnxb3_uCt_DNJ15kB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324689 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324691 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324694 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324698 INFO  (qtp1874391176-2566) [n:127.0.0.1:45993_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB_QigFZ_MSj0AoCFD0jyNev1oJIqnxb3_uCt_DNJ15kB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324699 INFO  (qtp1874391176-2566) [n:127.0.0.1:45993_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVnYuB_QigFZ_MSj0AoCFD0jyNev1oJIqnxb3_uCt_DNJ15kB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324705 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324707 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324710 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324711 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenVerify
   [junit4]   2> 324742 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationOtherOp
   [junit4]   2> 324748 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 324750 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63500/solr ready
   [junit4]   2> 324752 INFO  (qtp2044322013-2568) [n:127.0.0.1:61525_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params op=someSolrOperation&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 324752 INFO  (qtp2044322013-2568) [n:127.0.0.1:61525_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={op=someSolrOperation&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 324752 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=test&e=1485435662619&s=ADzN/NIid5s1N89U7xsXrFHhmlM="; Path=/; Domain=127.0.0.1; Expires=jue, 26-ene-2017 13:01:02 GMT; HttpOnly". Invalid 'expires' attribute: jue, 26-ene-2017 13:01:02 GMT
   [junit4]   2> 324753 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationOtherOp
   [junit4]   2> 324785 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenewFail
   [junit4]   2> 324790 WARN  (qtp2044322013-2575) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: foo tries to renew a token with renewer bar
   [junit4]   2> 324791 WARN  (qtp2044322013-2575) [    ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 324791 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324793 WARN  (qtp1874391176-2564) [n:127.0.0.1:45993_solr    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: foo tries to renew a token with renewer bar
   [junit4]   2> 324793 WARN  (qtp1874391176-2564) [n:127.0.0.1:45993_solr    ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 324793 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324807 WARN  (qtp2044322013-2571) [n:127.0.0.1:61525_solr    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 324807 WARN  (qtp2044322013-2571) [n:127.0.0.1:61525_solr    ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 324807 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324808 WARN  (qtp1874391176-2564) [n:127.0.0.1:45993_solr    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 324808 WARN  (qtp1874391176-2564) [n:127.0.0.1:45993_solr    ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 324808 WARN  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires=jue, 01-ene-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: jue, 01-ene-1970 00:00:00 GMT
   [junit4]   2> 324808 INFO  (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenewFail
   [junit4]   2> 324813 INFO  (jetty-launcher-307-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5bd8d11a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 324814 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1473568325
   [junit4]   2> 324822 INFO  (zkCallback-318-thread-2-processing-n:127.0.0.1:45993_solr) [n:127.0.0.1:45993_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 324826 INFO  (jetty-launcher-307-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6f2680d9{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 324829 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=57295353
   [junit4]   2> 324831 INFO  (jetty-launcher-307-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 324832 ERROR (Thread[Thread-1038,5,TGRP-TestSolrCloudWithDelegationTokens]) [n:127.0.0.1:61525_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 324845 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.c.Overseer Overseer (id=97347152134733829-127.0.0.1:45993_solr-n_0000000000) closing
   [junit4]   2> 324845 INFO  (jetty-launcher-307-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@72a039b1{/solr,null,UNAVAILABLE}
   [junit4]   2> 324845 INFO  (OverseerStateUpdate-97347152134733829-127.0.0.1:45993_solr-n_0000000000) [n:127.0.0.1:45993_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:45993_solr
   [junit4]   2> 324853 INFO  (jetty-launcher-307-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 324854 ERROR (Thread[Thread-1041,5,TGRP-TestSolrCloudWithDelegationTokens]) [n:127.0.0.1:45993_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 324862 INFO  (jetty-launcher-307-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3f3a2ffa{/solr,null,UNAVAILABLE}
   [junit4]   2> 324867 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:63500 63500
   [junit4]   2> 324925 INFO  (Thread-1021) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:63500 63500
   [junit4]   2> 324927 WARN  (Thread-1021) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	19	/solr/aliases.json
   [junit4]   2> 	19	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	19	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_7
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_6
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_9
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_5
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_12
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_11
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_10
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	19	/solr/live_nodes
   [junit4]   2> 	19	/solr/collections
   [junit4]   2> 	4	/solr/overseer/queue
   [junit4]   2> 	3	/solr/overseer/collection-queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 324929 INFO  (SUITE-TestSolrCloudWithDelegationTokens-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithDelegationTokens_D7CE5BBDD31F896E-001
   [junit4]   2> Jan 26, 2017 3:01:02 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=26, maxDocsPerChunk=7, blockSize=8), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=26, blockSize=8)), sim=RandomSimilarity(queryNorm=false): {}, locale=es-SV, timezone=Etc/GMT-1
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=59799208,total=144580608
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsRecoveryZkTest, OverseerStatusTest, TestInPlaceUpdatesDistrib, SuggesterFSTTest, BaseCdcrDistributedZkTest, IndexBasedSpellCheckerTest, TestCollationField, DocValuesNotIndexedTest, TestHashPartitioner, TestCloudPseudoReturnFields, TestFreeTextSuggestions, TestSubQueryTransformer, MBeansHandlerTest, TestGroupingSearch, TestImplicitCoreProperties, TestXmlQParserPlugin, TestGeoJSONResponseWriter, DistributedFacetPivotSmallTest, TestConfigSetImmutable, TestPartialUpdateDeduplication, SolrMetricsIntegrationTest, HardAutoCommitTest, SparseHLLTest, SpatialFilterTest, RecoveryZkTest, TestSolrQueryParserResource, DocValuesMultiTest, TestFieldTypeCollectionResource, TestCharFilters, ConcurrentDeleteAndCreateCollectionTest, HighlighterMaxOffsetTest, ConnectionManagerTest, TestRebalanceLeaders, TestRawResponseWriter, TestRandomDVFaceting, ClassificationUpdateProcessorFactoryTest, DistributedQueryElevationComponentTest, TestExpandComponent, MigrateRouteKeyTest, TestImpersonationWithHadoopAuth, DateFieldTest, BadCopyFieldTest, SolrGraphiteReporterTest, TestSolrCloudWithDelegationTokens]
   [junit4] Completed [91/684 (1!)] on J0 in 2.83s, 8 tests, 1 failure <<< FAILURES!

[...truncated 584 lines...]
   [junit4] Suite: org.apache.solr.cloud.PeerSyncReplicationTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/init-core-data-001
   [junit4]   2> 1090421 INFO  (SUITE-PeerSyncReplicationTest-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1090423 INFO  (SUITE-PeerSyncReplicationTest-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1090423 INFO  (SUITE-PeerSyncReplicationTest-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1090425 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1090425 INFO  (Thread-2728) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1090425 INFO  (Thread-2728) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1090525 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.ZkTestServer start zk server on port:38390
   [junit4]   2> 1090546 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1090550 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1090553 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1090555 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1090557 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1090560 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1090562 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1090564 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1090567 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1090570 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1090572 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1090768 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/control-001/cores/collection1
   [junit4]   2> 1090769 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1090771 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@dfb45e9{/,null,AVAILABLE}
   [junit4]   2> 1090774 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4196ced0{HTTP/1.1,[http/1.1]}{127.0.0.1:40517}
   [junit4]   2> 1090774 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.Server Started @1094167ms
   [junit4]   2> 1090774 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/tempDir-001/control/data, hostContext=/, hostPort=40517, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/control-001/cores}
   [junit4]   2> 1090774 ERROR (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1090774 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1090774 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1090774 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1090774 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-26T03:13:48.642Z
   [junit4]   2> 1090779 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1090779 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/control-001/solr.xml
   [junit4]   2> 1090786 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38390/solr
   [junit4]   2> 1090830 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:40517_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40517_
   [junit4]   2> 1090832 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:40517_    ] o.a.s.c.Overseer Overseer (id=97347202468675588-127.0.0.1:40517_-n_0000000000) starting
   [junit4]   2> 1090846 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:40517_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40517_
   [junit4]   2> 1090848 INFO  (OverseerStateUpdate-97347202468675588-127.0.0.1:40517_-n_0000000000) [n:127.0.0.1:40517_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1090906 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:40517_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/control-001/cores
   [junit4]   2> 1090906 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:40517_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1090913 INFO  (OverseerStateUpdate-97347202468675588-127.0.0.1:40517_-n_0000000000) [n:127.0.0.1:40517_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1091928 WARN  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1091929 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1091948 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1092026 WARN  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1092028 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1092045 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1092046 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/control-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/control-001/cores/collection1/data/]
   [junit4]   2> 1092046 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4733d4be
   [junit4]   2> 1092048 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=587178138, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1092062 WARN  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1092072 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1092072 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1092078 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1092078 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1092079 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=63.2822265625, floorSegmentMB=1.58203125, forceMergeDeletesPctAllowed=8.527895922583065, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1092080 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6eba76a1[collection1] main]
   [junit4]   2> 1092081 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1092082 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1092082 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1092083 INFO  (searcherExecutor-3730-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6eba76a1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1092083 INFO  (coreLoadExecutor-3729-thread-1-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557555241236299776
   [junit4]   2> 1092095 INFO  (coreZkRegister-3722-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1092095 INFO  (coreZkRegister-3722-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1092095 INFO  (coreZkRegister-3722-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40517/collection1/
   [junit4]   2> 1092095 INFO  (coreZkRegister-3722-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1092095 INFO  (coreZkRegister-3722-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40517/collection1/ has no replicas
   [junit4]   2> 1092102 INFO  (coreZkRegister-3722-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40517/collection1/ shard1
   [junit4]   2> 1092190 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1092192 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38390/solr ready
   [junit4]   2> 1092192 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1092255 INFO  (coreZkRegister-3722-thread-1-processing-n:127.0.0.1:40517_ x:collection1 c:control_collection) [n:127.0.0.1:40517_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1092340 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-1-001/cores/collection1
   [junit4]   2> 1092340 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-1-001
   [junit4]   2> 1092341 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1092343 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@f633f96{/,null,AVAILABLE}
   [junit4]   2> 1092343 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@543e662e{HTTP/1.1,[http/1.1]}{127.0.0.1:49682}
   [junit4]   2> 1092343 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.Server Started @1095737ms
   [junit4]   2> 1092343 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=49682, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-1-001/cores}
   [junit4]   2> 1092343 ERROR (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1092344 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1092344 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1092344 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1092344 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-26T03:13:50.212Z
   [junit4]   2> 1092348 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1092348 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-1-001/solr.xml
   [junit4]   2> 1092354 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38390/solr
   [junit4]   2> 1092370 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:49682_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1092379 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:49682_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49682_
   [junit4]   2> 1092382 INFO  (zkCallback-1336-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1092383 INFO  (zkCallback-1341-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1092383 INFO  (zkCallback-1332-thread-4-processing-n:127.0.0.1:40517_) [n:127.0.0.1:40517_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1092427 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:49682_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-1-001/cores
   [junit4]   2> 1092427 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [n:127.0.0.1:49682_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1092431 INFO  (OverseerStateUpdate-97347202468675588-127.0.0.1:40517_-n_0000000000) [n:127.0.0.1:40517_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1093446 WARN  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1093446 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1093460 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1093537 WARN  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1093538 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1093555 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1093555 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-1-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 1093556 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4733d4be
   [junit4]   2> 1093558 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=587178138, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1093565 WARN  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1093580 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1093580 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1093581 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1093581 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1093581 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=63.2822265625, floorSegmentMB=1.58203125, forceMergeDeletesPctAllowed=8.527895922583065, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1093582 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6e6ad71f[collection1] main]
   [junit4]   2> 1093584 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1093585 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1093585 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1093586 INFO  (searcherExecutor-3741-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6e6ad71f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1093586 INFO  (coreLoadExecutor-3740-thread-1-processing-n:127.0.0.1:49682_) [n:127.0.0.1:49682_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557555242812309504
   [junit4]   2> 1093596 INFO  (coreZkRegister-3735-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1093596 INFO  (coreZkRegister-3735-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1093596 INFO  (coreZkRegister-3735-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:49682/collection1/
   [junit4]   2> 1093596 INFO  (coreZkRegister-3735-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1093596 INFO  (coreZkRegister-3735-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:49682/collection1/ has no replicas
   [junit4]   2> 1093603 INFO  (coreZkRegister-3735-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:49682/collection1/ shard1
   [junit4]   2> 1093754 INFO  (coreZkRegister-3735-thread-1-processing-n:127.0.0.1:49682_ x:collection1 c:collection1) [n:127.0.0.1:49682_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1093841 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-2-001/cores/collection1
   [junit4]   2> 1093842 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-2-001
   [junit4]   2> 1093842 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1093844 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15fc6839{/,null,AVAILABLE}
   [junit4]   2> 1093844 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3f242117{HTTP/1.1,[http/1.1]}{127.0.0.1:52344}
   [junit4]   2> 1093844 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.Server Started @1097239ms
   [junit4]   2> 1093844 INFO  (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=52344, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_D7CE5BBDD31F896E-001/shard-2-001/cores}
   [junit4]   2> 1093844 ERROR (TEST-PeerSyncReplicationTest.test-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1093849 INFO  (TEST-PeerSyncReplic

[...truncated too long message...]

r Overseer Loop exiting : 127.0.0.1:52039_
   [junit4]   2> 3532338 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x159d8e850d20012, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3532340 INFO  (zkCallback-3243-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 3532346 INFO  (TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 3532347 INFO  (TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@75c4e653{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 3532347 INFO  (TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[D7CE5BBDD31F896E]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2b0525d{/,null,UNAVAILABLE}
   [junit4]   2> 3532351 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x159d8e850d20005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3532351 INFO  (TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 61102
   [junit4]   2> 3532358 INFO  (TEST-ShardSplitTest.testSplitAfterFailedSplit-seed#[D7CE5BBDD31F896E]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34641 34641
   [junit4]   2> 3537387 INFO  (Thread-5825) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34641 34641
   [junit4]   2> 3537389 WARN  (Thread-5825) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	5	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	5	/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	78	/solr/overseer/queue
   [junit4]   2> 	31	/solr/overseer/collection-queue-work
   [junit4]   2> 	18	/solr/overseer/queue-work
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 
   [junit4]   2> 3537391 INFO  (SUITE-ShardSplitTest-seed#[D7CE5BBDD31F896E]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.ShardSplitTest_D7CE5BBDD31F896E-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=BlockTreeOrds(blocksize=128), shard_s=PostingsFormat(name=MockRandom), n_ti=PostingsFormat(name=MockRandom), a_t=PostingsFormat(name=MockRandom), a_i=BlockTreeOrds(blocksize=128), id=BlockTreeOrds(blocksize=128)}, docValues:{rnd_b=DocValuesFormat(name=Lucene70), shard_s=DocValuesFormat(name=Direct), _version_=DocValuesFormat(name=Direct), n_ti=DocValuesFormat(name=Direct), a_t=DocValuesFormat(name=Direct), a_i=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=899, maxMBSortInHeap=6.530966808429264, sim=RandomSimilarity(queryNorm=true): {}, locale=it, timezone=Australia/South
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=205600976,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsRecoveryZkTest, OverseerStatusTest, TestInPlaceUpdatesDistrib, SuggesterFSTTest, BaseCdcrDistributedZkTest, IndexBasedSpellCheckerTest, TestCollationField, DocValuesNotIndexedTest, TestHashPartitioner, TestCloudPseudoReturnFields, TestFreeTextSuggestions, TestSubQueryTransformer, MBeansHandlerTest, TestGroupingSearch, TestImplicitCoreProperties, TestXmlQParserPlugin, TestGeoJSONResponseWriter, DistributedFacetPivotSmallTest, TestConfigSetImmutable, TestPartialUpdateDeduplication, SolrMetricsIntegrationTest, HardAutoCommitTest, SparseHLLTest, SpatialFilterTest, RecoveryZkTest, TestSolrQueryParserResource, DocValuesMultiTest, TestFieldTypeCollectionResource, TestCharFilters, ConcurrentDeleteAndCreateCollectionTest, HighlighterMaxOffsetTest, ConnectionManagerTest, TestRebalanceLeaders, TestRawResponseWriter, TestRandomDVFaceting, ClassificationUpdateProcessorFactoryTest, DistributedQueryElevationComponentTest, TestExpandComponent, MigrateRouteKeyTest, TestImpersonationWithHadoopAuth, DateFieldTest, BadCopyFieldTest, SolrGraphiteReporterTest, TestSolrCloudWithDelegationTokens, OverriddenZkACLAndCredentialsProvidersTest, TestTolerantUpdateProcessorCloud, CloudMLTQParserTest, IndexSchemaRuntimeFieldTest, ZkStateWriterTest, DistributedQueueTest, ManagedSchemaRoundRobinCloudTest, HttpPartitionTest, TestCursorMarkWithoutUniqueKey, DeleteReplicaTest, TestGraphTermsQParserPlugin, CustomCollectionTest, HdfsSyncSliceTest, TestDownShardTolerantSearch, TestCollapseQParserPlugin, HdfsTlogReplayBufferedWhileIndexingTest, TestRequestForwarding, TestBackupRepositoryFactory, TestFieldCacheWithThreads, TestCSVResponseWriter, HdfsDirectoryFactoryTest, DirectoryFactoryTest, HdfsChaosMonkeySafeLeaderTest, TestFieldCacheSortRandom, SolrRequestParserTest, ClassificationUpdateProcessorTest, SolrTestCaseJ4Test, TestReplicationHandler, FileUtilsTest, TestMiniSolrCloudCluster, RemoteQueryErrorTest, TestCustomStream, PeerSyncTest, TestSolrJ, TestSchemaSimilarityResource, TestSchemaNameResource, ZkNodePropsTest, TestLeaderElectionZkExpiry, TestSurroundQueryParser, TestFastLRUCache, RequiredFieldsTest, TimeZoneUtilsTest, SegmentsInfoRequestHandlerTest, TestDocumentBuilder, UnloadDistributedZkTest, SynonymTokenizerTest, SolrCloudExampleTest, ActionThrottleTest, TestFaceting, CdcrRequestHandlerTest, LeaderFailoverAfterPartitionTest, TestManagedResourceStorage, ZkCLITest, QueryEqualityTest, TestJettySolrRunner, MultiTermTest, UtilsToolTest, AddSchemaFieldsUpdateProcessorFactoryTest, DirectUpdateHandlerTest, SharedFSAutoReplicaFailoverTest, BinaryUpdateRequestHandlerTest, TestAddFieldRealTimeGet, TestDeleteCollectionOnDownNodes, TestTrieFacet, SolrIndexConfigTest, TestClusterProperties, TestReloadAndDeleteDocs, TestStressVersions, PeerSyncReplicationTest, DOMUtilTest, TestInitParams, TestStandardQParsers, TestFieldSortValues, TestSolrCLIRunExample, TestCopyFieldCollectionResource, TestSolrDeletionPolicy1, TestBlobHandler, TestComplexPhraseQParserPlugin, TestUseDocValuesAsStored, AlternateDirectoryTest, TestBlendedInfixSuggestions, TestCloudSchemaless, BlobRepositoryCloudTest, SpatialHeatmapFacetsTest, SampleTest, TestXIncludeConfig, TestObjectReleaseTracker, PluginInfoTest, PKIAuthenticationIntegrationTest, TlogReplayBufferedWhileIndexingTest, DistributedMLTComponentTest, DistributedFacetExistsSmallTest, DirectSolrConnectionTest, ChaosMonkeyNothingIsSafeTest, DistributedFacetPivotLongTailTest, TestFunctionQuery, TestReqParamsAPI, SimpleCollectionCreateDeleteTest, SimplePostToolTest, TestInitQParser, SOLR749Test, ChangedSchemaMergeTest, HdfsNNFailoverTest, TestShortCircuitedRequests, TestManagedSchema, TestExtendedDismaxParser, CurrencyFieldXmlFileTest, SpatialRPTFieldTypeTest, TestRandomFlRTGCloud, SolrMetricManagerTest, TestSort, TestMacros, TestCustomDocTransformer, TestHighlightDedupGrouping, DisMaxRequestHandlerTest, BasicAuthIntegrationTest, SchemaVersionSpecificBehaviorTest, AnalysisAfterCoreReloadTest, XmlUpdateRequestHandlerTest, VMParamsZkACLAndCredentialsProvidersTest, TestNumericTerms32, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, SyncSliceTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, ShardRoutingCustomTest, TestDistributedSearch, ZkControllerTest, TestJoin, TestCoreContainer, SimpleFacetsTest, SolrCoreTest, SolrCmdDistributorTest, QueryElevationComponentTest, TestLazyCores, HighlighterTest, SoftAutoCommitTest, ShowFileRequestHandlerTest, SolrIndexSplitterTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, CoreAdminHandlerTest, DocValuesTest, SuggesterTSTTest, SuggesterTest, TestStressLucene, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, StatelessScriptUpdateProcessorFactoryTest, SolrInfoMBeanTest, CacheHeaderTest, LukeRequestHandlerTest, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, RequestHandlersTest, TestValueSourceCache, MoreLikeThisHandlerTest, RegexBoostProcessorTest, TestJmxIntegration, TestAnalyzedSuggestions, JsonLoaderTest, SearchHandlerTest, TestLFUCache, DistributedIntervalFacetingTest, TestCrossCoreJoin, TestTolerantSearch, TestEmbeddedSolrServerConstructors, AliasIntegrationTest, AssignTest, AsyncCallRequestStatusResponseTest, CdcrBootstrapTest, ConfigSetsAPITest, DeleteInactiveReplicaTest, DeleteShardTest, DistribCursorPagingTest, MultiThreadedOCPTest, RecoveryAfterSoftCommitTest, SaslZkACLProviderTest, ShardSplitTest]
   [junit4] Completed [515/684 (3!)] on J0 in 605.93s, 5 tests, 1 failure <<< FAILURES!

[...truncated 62987 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1097 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1097/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 27494 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/jdk/instances/jdk1.8.0/jre/bin/java -XX:+UseCompressedOops -XX:+UseSerialGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=95339D88742FFB44 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.0.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp -Dcommon.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene -Dclover.db.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/clover/db -Djava.security.policy=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=7.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Djunit4.childvm.cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0 -Djunit4.childvm.id=0 -Djunit4.childvm.count=2 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=US-ASCII -classpath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/test-framework/lib/junit4-ant-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/common/lucene-analyzers-common-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/lucene-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/backward-codecs/lucene-backward-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/highlighter/lucene-highlighter-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/memory/lucene-memory-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/misc/lucene-misc-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/spatial-extras/lucene-spatial-extras-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/expressions/lucene-expressions-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/suggest/lucene-suggest-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/grouping/lucene-grouping-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queries/lucene-queries-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/lucene-queryparser-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/join/lucene-join-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/sandbox/lucene-sandbox-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/classification/lucene-classification-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-commons-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/caffeine-1.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-cli-1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-codec-1.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-configuration-1.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-exec-1.3.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-fileupload-1.3.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-lang-2.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-client-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-framework-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-recipes-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/dom4j-1.6.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/guava-14.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-annotations-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-auth-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-common-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-hdfs-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hppc-0.7.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-dataformat-smile-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-mapper-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/joda-time-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/log4j-1.2.17.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/protobuf-java-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slf4j-log4j12-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/spatial4j-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/t-digest-3.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpclient-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpcore-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpmime-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/noggit-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/slf4j-api-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/stax2-api-3.1.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/zookeeper-3.4.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/gmetric4j-1.0.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/javax.servlet-api-3.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-continuation-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-deploy-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-http-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-io-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-jmx-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-rewrite-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-security-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-server-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlet-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlets-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-util-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-webapp-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-xml-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-core-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-ganglia-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-graphite-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jetty9-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jvm-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/junit-4.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/randomizedtesting-runner-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr-2.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/api-all-1.0.0-M20.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/bcprov-jdk15-1.45.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/byte-buddy-1.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/cglib-nodep-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-math3-3.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/easymock-3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/ehcache-core-2.4.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-annotations-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-databind-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-core-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-server-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-util-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mina-core-2.0.0-M5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mockito-core-2.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/netty-all-4.0.36.Final.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/objenesis-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/contrib/solr-analysis-extras/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/export/home/jenkins/.ant/lib/ivy-2.3.0.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/export/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.0.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J0-20170125_183804_9486969975056441036190.events @/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J0-20170125_183804_9481873891887823359532.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: java.io.IOException: No space left on device
   [junit4] 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
   [junit4] 	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
   [junit4] 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
   [junit4] 	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
   [junit4] 	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
   [junit4] 	at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
   [junit4] 	at java.nio.channels.Channels.writeFully(Channels.java:101)
   [junit4] 	at java.nio.channels.Channels.access$000(Channels.java:61)
   [junit4] 	at java.nio.channels.Channels$1.write(Channels.java:174)
   [junit4] 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4] 	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4] 	at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.io.Closer.close(Closer.java:214)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1526)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/jdk/instances/jdk1.8.0/jre/bin/java -XX:+UseCompressedOops -XX:+UseSerialGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=95339D88742FFB44 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.0.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp -Dcommon.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene -Dclover.db.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/clover/db -Djava.security.policy=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=7.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Djunit4.childvm.cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=2 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=US-ASCII -classpath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/test-framework/lib/junit4-ant-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/common/lucene-analyzers-common-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/lucene-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/backward-codecs/lucene-backward-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/highlighter/lucene-highlighter-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/memory/lucene-memory-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/misc/lucene-misc-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/spatial-extras/lucene-spatial-extras-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/expressions/lucene-expressions-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/suggest/lucene-suggest-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/grouping/lucene-grouping-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queries/lucene-queries-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/lucene-queryparser-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/join/lucene-join-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/sandbox/lucene-sandbox-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/classification/lucene-classification-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-commons-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/caffeine-1.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-cli-1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-codec-1.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-configuration-1.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-exec-1.3.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-fileupload-1.3.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-lang-2.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-client-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-framework-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-recipes-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/dom4j-1.6.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/guava-14.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-annotations-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-auth-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-common-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-hdfs-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hppc-0.7.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-dataformat-smile-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-mapper-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/joda-time-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/log4j-1.2.17.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/protobuf-java-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slf4j-log4j12-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/spatial4j-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/t-digest-3.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpclient-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpcore-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpmime-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/noggit-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/slf4j-api-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/stax2-api-3.1.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/zookeeper-3.4.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/gmetric4j-1.0.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/javax.servlet-api-3.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-continuation-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-deploy-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-http-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-io-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-jmx-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-rewrite-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-security-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-server-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlet-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlets-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-util-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-webapp-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-xml-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-core-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-ganglia-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-graphite-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jetty9-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jvm-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/junit-4.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/randomizedtesting-runner-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr-2.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/api-all-1.0.0-M20.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/bcprov-jdk15-1.45.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/byte-buddy-1.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/cglib-nodep-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-math3-3.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/easymock-3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/ehcache-core-2.4.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-annotations-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-databind-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-core-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-server-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-util-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mina-core-2.0.0-M5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mockito-core-2.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/netty-all-4.0.36.Final.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/objenesis-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/contrib/solr-analysis-extras/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/export/home/jenkins/.ant/lib/ivy-2.3.0.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/export/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.0.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J1-20170125_183804_9483651502784676349126.events @/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J1-20170125_183804_948117337208032958769.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: java.io.IOException: No space left on device
   [junit4] 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
   [junit4] 	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
   [junit4] 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
   [junit4] 	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
   [junit4] 	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
   [junit4] 	at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
   [junit4] 	at java.nio.channels.Channels.writeFully(Channels.java:101)
   [junit4] 	at java.nio.channels.Channels.access$000(Channels.java:61)
   [junit4] 	at java.nio.channels.Channels$1.write(Channels.java:174)
   [junit4] 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4] 	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4] 	at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.io.Closer.close(Closer.java:214)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1526)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:775: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:719: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:252: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:536: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1445: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:985: At least one slave process threw an exception, first: No space left on device

Total time: 49 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1096 - Failure!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1096/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 13189 lines...]
   [junit4] JVM J0: stderr was not empty, see: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J0-20170125_063949_6198742212180168337133.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] WARN: Unhandled exception in event serialization. -> java.security.PrivilegedActionException: java.io.IOException: No space left on device
   [junit4] 	at java.security.AccessController.doPrivileged(Native Method)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.events.Serializer.flushQueue(Serializer.java:114)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:99)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$3$2.write(SlaveMain.java:459)
   [junit4] 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4] 	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4] 	at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4] 	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
   [junit4] 	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
   [junit4] 	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
   [junit4] 	at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
   [junit4] 	at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
   [junit4] 	at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
   [junit4] 	at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
   [junit4] 	at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
   [junit4] 	at org.apache.log4j.Category.callAppenders(Category.java:206)
   [junit4] 	at org.apache.log4j.Category.forcedLog(Category.java:391)
   [junit4] 	at org.apache.log4j.Category.log(Category.java:856)
   [junit4] 	at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
   [junit4] 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2407)
   [junit4] 	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
   [junit4] 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
   [junit4] 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
   [junit4] 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
   [junit4] 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4] 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
   [junit4] 
   [junit4] 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4] 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4] 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4] 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4] 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4] 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4] 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4] 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4] 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:395)
   [junit4] 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4] 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4] 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4] 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4] 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4] 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4] 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4] 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4] 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4] 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4] 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4] 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Caused by: java.io.IOException: No space left on device
   [junit4] 	at java.io.FileOutputStream.writeBytes(Native Method)
   [junit4] 	at java.io.FileOutputStream.write(FileOutputStream.java:326)
   [junit4] 	at java.io.BufferedOutputStre
   [junit4] am.write(BufferedOutputStream.java:122)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
   [junit4] 	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4] 	at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
   [junit4] 	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
   [junit4] 	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
   [junit4] 	at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.string(JsonWriter.java:572)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.gson.stream.JsonWriter.value(JsonWriter.java:415)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.events.AbstractEvent.writeBinaryProperty(AbstractEvent.java:36)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.events.AppendStdErrEvent.serialize(AppendStdErrEvent.java:30)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:119)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$2.run(Serializer.java:114)
   [junit4] 	... 48 more
   [junit4] <<< JVM J0: EOF ----

[...truncated 1 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/jdk/instances/jdk1.8.0/jre/bin/java -XX:+UseCompressedOops -XX:+UseSerialGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=92680163FFACB9CF -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.0.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp -Dcommon.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene -Dclover.db.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/clover/db -Djava.security.policy=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=7.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Djunit4.childvm.cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0 -Djunit4.childvm.id=0 -Djunit4.childvm.count=2 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/test-framework/lib/junit4-ant-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/common/lucene-analyzers-common-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/lucene-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/backward-codecs/lucene-backward-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/highlighter/lucene-highlighter-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/memory/lucene-memory-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/misc/lucene-misc-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/spatial-extras/lucene-spatial-extras-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/expressions/lucene-expressions-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/suggest/lucene-suggest-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/grouping/lucene-grouping-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queries/lucene-queries-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/lucene-queryparser-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/join/lucene-join-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/sandbox/lucene-sandbox-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/classification/lucene-classification-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-commons-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/caffeine-1.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-cli-1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-codec-1.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-configuration-1.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-exec-1.3.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-fileupload-1.3.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-lang-2.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-client-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-framework-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-recipes-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/dom4j-1.6.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/guava-14.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-annotations-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-auth-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-common-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-hdfs-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hppc-0.7.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-dataformat-smile-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-mapper-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/joda-time-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/log4j-1.2.17.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/protobuf-java-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slf4j-log4j12-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/spatial4j-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/t-digest-3.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpclient-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpcore-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpmime-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/noggit-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/slf4j-api-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/stax2-api-3.1.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/zookeeper-3.4.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/gmetric4j-1.0.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/javax.servlet-api-3.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-continuation-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-deploy-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-http-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-io-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-jmx-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-rewrite-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-security-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-server-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlet-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlets-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-util-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-webapp-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-xml-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-core-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-ganglia-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-graphite-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jetty9-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jvm-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/junit-4.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/randomizedtesting-runner-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr-2.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/api-all-1.0.0-M20.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/bcprov-jdk15-1.45.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/byte-buddy-1.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/cglib-nodep-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-math3-3.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/easymock-3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/ehcache-core-2.4.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-annotations-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-databind-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-core-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-server-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-util-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mina-core-2.0.0-M5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mockito-core-2.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/netty-all-4.0.36.Final.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/objenesis-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/contrib/solr-analysis-extras/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/export/home/jenkins/.ant/lib/ivy-2.3.0.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/export/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.0.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J0-20170125_063949_6198036749905368536198.events @/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J0-20170125_063949_6194063787431118156367.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1540)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/jdk/instances/jdk1.8.0/jre/bin/java -XX:+UseCompressedOops -XX:+UseSerialGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=92680163FFACB9CF -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.0.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp -Dcommon.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene -Dclover.db.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/clover/db -Djava.security.policy=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=7.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Djunit4.childvm.cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1 -Djunit4.childvm.id=1 -Djunit4.childvm.count=2 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/test-framework/lib/junit4-ant-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/test-framework/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/common/lucene-analyzers-common-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/codecs/lucene-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/backward-codecs/lucene-backward-codecs-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/highlighter/lucene-highlighter-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/memory/lucene-memory-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/misc/lucene-misc-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/spatial-extras/lucene-spatial-extras-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/expressions/lucene-expressions-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/suggest/lucene-suggest-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/grouping/lucene-grouping-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queries/lucene-queries-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/lucene-queryparser-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/join/lucene-join-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/sandbox/lucene-sandbox-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/classification/lucene-classification-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/asm-commons-5.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/caffeine-1.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-cli-1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-codec-1.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-configuration-1.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-exec-1.3.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-fileupload-1.3.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/commons-lang-2.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-client-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-framework-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/curator-recipes-2.8.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/dom4j-1.6.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/guava-14.0.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-annotations-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-auth-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-common-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hadoop-hdfs-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/hppc-0.7.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-core-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-dataformat-smile-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jackson-mapper-asl-1.9.13.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/joda-time-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/log4j-1.2.17.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/protobuf-java-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slf4j-log4j12-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/spatial4j-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/lib/t-digest-3.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/commons-io-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpclient-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpcore-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/httpmime-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/jcl-over-slf4j-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/noggit-0.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/slf4j-api-1.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/stax2-api-3.1.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/solrj/lib/zookeeper-3.4.6.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/gmetric4j-1.0.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/javax.servlet-api-3.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-continuation-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-deploy-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-http-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-io-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-jmx-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-rewrite-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-security-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-server-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlet-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-servlets-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-util-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-webapp-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/jetty-xml-9.3.14.v20161028.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-core-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-ganglia-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-graphite-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jetty9-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/server/lib/metrics-jvm-3.1.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/core/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/junit-4.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/test-framework/lib/randomizedtesting-runner-2.5.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr-2.7.7.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/api-all-1.0.0-M20.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/bcprov-jdk15-1.45.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/byte-buddy-1.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/cglib-nodep-2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-collections-3.2.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/commons-math3-3.4.1.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/easymock-3.0.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/ehcache-core-2.4.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-annotations-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jackson-databind-2.5.4.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-core-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jersey-server-1.9.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/jetty-util-6.1.26.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mina-core-2.0.0-M5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/mockito-core-2.6.2.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/netty-all-4.0.36.Final.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/objenesis-2.5.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/presto-parser-0.122.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/test-lib/slice-0.10.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/contrib/solr-analysis-extras/classes/java:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build/queryparser/classes/test:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/export/home/jenkins/.ant/lib/ivy-2.3.0.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/usr/jdk/instances/jdk1.8.0/lib/tools.jar:/export/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.5.0.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J1-20170125_063949_6198848398130735396871.events @/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp/junit4-J1-20170125_063949_6193965476755137824447.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: java.io.IOException: No space left on device
   [junit4] 	at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
   [junit4] 	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
   [junit4] 	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
   [junit4] 	at sun.nio.ch.IOUtil.write(IOUtil.java:65)
   [junit4] 	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
   [junit4] 	at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
   [junit4] 	at java.nio.channels.Channels.writeFully(Channels.java:101)
   [junit4] 	at java.nio.channels.Channels.access$000(Channels.java:61)
   [junit4] 	at java.nio.channels.Channels$1.write(Channels.java:174)
   [junit4] 	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4] 	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4] 	at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.io.Closer.close(Closer.java:214)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1526)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:126)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:982)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:979)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:775: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:719: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:252: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:536: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1445: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:985: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 58 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1095 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1095/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
	at __randomizedtesting.SeedInfo.seed([2E9385B7DD1AD69A:462CB09D0D80C476]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.cancelDelegationToken(TestDelegationWithHadoopAuth.java:128)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenCancelFail(TestDelegationWithHadoopAuth.java:290)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12735 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_2E9385B7DD1AD69A-001/init-core-data-001
   [junit4]   2> 2967728 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[2E9385B7DD1AD69A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2967728 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[2E9385B7DD1AD69A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2967729 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[2E9385B7DD1AD69A]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_2E9385B7DD1AD69A-001/tempDir-001
   [junit4]   2> 2967729 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[2E9385B7DD1AD69A]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2967729 INFO  (Thread-10282) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2967729 INFO  (Thread-10282) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2967829 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[2E9385B7DD1AD69A]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:46204
   [junit4]   2> 2967839 INFO  (jetty-launcher-4539-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2967839 INFO  (jetty-launcher-4539-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2967842 INFO  (jetty-launcher-4539-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4568e2ff{/solr,null,AVAILABLE}
   [junit4]   2> 2967843 INFO  (jetty-launcher-4539-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1231c036{/solr,null,AVAILABLE}
   [junit4]   2> 2967843 INFO  (jetty-launcher-4539-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@34589e44{HTTP/1.1,[http/1.1]}{127.0.0.1:58313}
   [junit4]   2> 2967844 INFO  (jetty-launcher-4539-thread-2) [    ] o.e.j.s.Server Started @2972252ms
   [junit4]   2> 2967844 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58313}
   [junit4]   2> 2967844 INFO  (jetty-launcher-4539-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@de21a94{HTTP/1.1,[http/1.1]}{127.0.0.1:38032}
   [junit4]   2> 2967844 INFO  (jetty-launcher-4539-thread-1) [    ] o.e.j.s.Server Started @2972252ms
   [junit4]   2> 2967844 ERROR (jetty-launcher-4539-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2967844 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2967844 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38032}
   [junit4]   2> 2967844 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2967845 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2967845 ERROR (jetty-launcher-4539-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2967845 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-24T20:23:07.788Z
   [junit4]   2> 2967845 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 2967845 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2967845 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2967845 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-24T20:23:07.788Z
   [junit4]   2> 2967848 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2967848 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2967865 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46204/solr
   [junit4]   2> 2967865 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46204/solr
   [junit4]   2> 2967910 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38032_solr
   [junit4]   2> 2967911 INFO  (jetty-launcher-4539-thread-2) [n:127.0.0.1:58313_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58313_solr
   [junit4]   2> 2967911 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.s.c.Overseer Overseer (id=97339925310603269-127.0.0.1:38032_solr-n_0000000000) starting
   [junit4]   2> 2967924 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38032_solr
   [junit4]   2> 2967937 INFO  (zkCallback-4549-thread-1-processing-n:127.0.0.1:38032_solr) [n:127.0.0.1:38032_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2967937 INFO  (zkCallback-4550-thread-1-processing-n:127.0.0.1:58313_solr) [n:127.0.0.1:58313_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2967939 INFO  (zkCallback-4550-thread-2-processing-n:127.0.0.1:58313_solr) [n:127.0.0.1:58313_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2967941 INFO  (zkCallback-4549-thread-2-processing-n:127.0.0.1:38032_solr) [n:127.0.0.1:38032_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2968019 INFO  (jetty-launcher-4539-thread-2) [n:127.0.0.1:58313_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 2968032 INFO  (jetty-launcher-4539-thread-2) [n:127.0.0.1:58313_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 2968087 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 2968087 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 2968132 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2968132 INFO  (jetty-launcher-4539-thread-2) [n:127.0.0.1:58313_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2968160 INFO  (jetty-launcher-4539-thread-1-EventThread) [n:127.0.0.1:38032_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2968160 INFO  (jetty-launcher-4539-thread-2-EventThread) [n:127.0.0.1:58313_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2968293 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 2968299 INFO  (jetty-launcher-4539-thread-1) [n:127.0.0.1:38032_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_2E9385B7DD1AD69A-001/tempDir-001/node1/.
   [junit4]   2> 2968308 INFO  (jetty-launcher-4539-thread-2) [n:127.0.0.1:58313_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 2968310 INFO  (jetty-launcher-4539-thread-2) [n:127.0.0.1:58313_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_2E9385B7DD1AD69A-001/tempDir-001/node2/.
   [junit4]   2> 2969004 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancelFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancelFail
   [junit4]   2> 2969087 ERROR (qtp1824882325-29525) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_1]
   [junit4]   2> 2969089 ERROR (qtp1321832301-29536) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_1]
   [junit4]   2> 2969104 ERROR (qtp1824882325-29528) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_2]
   [junit4]   2> 2969107 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancelFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancelFail
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDelegationWithHadoopAuth -Dtests.method=testDelegationTokenCancelFail -Dtests.seed=2E9385B7DD1AD69A -Dtests.slow=true -Dtests.locale=ms-MY -Dtests.timezone=America/Regina -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.14s J1 | TestDelegationWithHadoopAuth.testDelegationTokenCancelFail <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<200> but was:<404>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E9385B7DD1AD69A:462CB09D0D80C476]:0)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.cancelDelegationToken(TestDelegationWithHadoopAuth.java:128)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenCancelFail(TestDelegationWithHadoopAuth.java:290)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2969147 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenew
   [junit4]   2> 2969171 INFO  (qtp1321832301-29538) [n:127.0.0.1:38032_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXwjigFZ9jIAIwMDFJPaj3KJBP1FA_aLV1NcT4YzL5G1B3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969171 INFO  (qtp1321832301-29538) [n:127.0.0.1:38032_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXwjigFZ9jIAIwMDFJPaj3KJBP1FA_aLV1NcT4YzL5G1B3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969182 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969191 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969193 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969196 INFO  (qtp1824882325-29528) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXwjigFZ9jIAIwMDFJPaj3KJBP1FA_aLV1NcT4YzL5G1B3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969196 INFO  (qtp1824882325-29528) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXwjigFZ9jIAIwMDFJPaj3KJBP1FA_aLV1NcT4YzL5G1B3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969202 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969230 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969232 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969233 INFO  (qtp1824882325-29527) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXxaigFZ9jIAWgQDFNPvr27nBf7YXyglVyB4HN3yQ_HAB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969233 INFO  (qtp1824882325-29527) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXxaigFZ9jIAWgQDFNPvr27nBf7YXyglVyB4HN3yQ_HAB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969239 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969245 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969247 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969248 INFO  (qtp1824882325-29531) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXxaigFZ9jIAWgQDFNPvr27nBf7YXyglVyB4HN3yQ_HAB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969248 INFO  (qtp1824882325-29531) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXxaigFZ9jIAWgQDFNPvr27nBf7YXyglVyB4HN3yQ_HAB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969252 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969266 INFO  (qtp1321832301-29526) [n:127.0.0.1:38032_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyMigFZ9jIAjAUDFIHbdFkZE2K2P9RTO9zJXkrWQWQtB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969266 INFO  (qtp1321832301-29526) [n:127.0.0.1:38032_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyMigFZ9jIAjAUDFIHbdFkZE2K2P9RTO9zJXkrWQWQtB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969269 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969271 INFO  (qtp1824882325-29527) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyMigFZ9jIAjAUDFIHbdFkZE2K2P9RTO9zJXkrWQWQtB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969271 INFO  (qtp1824882325-29527) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyMigFZ9jIAjAUDFIHbdFkZE2K2P9RTO9zJXkrWQWQtB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969274 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969293 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969295 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969296 INFO  (qtp1824882325-29523) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyiigFZ9jIAogYDFDogflVJyB87RBfNN82mZK4WxIZEB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969296 INFO  (qtp1824882325-29523) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyiigFZ9jIAogYDFDogflVJyB87RBfNN82mZK4WxIZEB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969302 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969303 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969305 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969311 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969312 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969313 INFO  (qtp1824882325-29525) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyiigFZ9jIAogYDFDogflVJyB87RBfNN82mZK4WxIZEB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969313 INFO  (qtp1824882325-29525) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJXyiigFZ9jIAogYDFDogflVJyB87RBfNN82mZK4WxIZEB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969317 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969317 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenew
   [junit4]   2> 2969346 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancel
   [junit4]   2> 2969362 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969363 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969365 ERROR (qtp1824882325-29531) [n:127.0.0.1:58313_solr    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_7]
   [junit4]   2> 2969374 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969376 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969388 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969391 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969392 ERROR (qtp1824882325-29531) [n:127.0.0.1:58313_solr    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_8]
   [junit4]   2> 2969400 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969402 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969403 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969409 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969411 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969412 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969413 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancel
   [junit4]   2> 2969443 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationOtherOp
   [junit4]   2> 2969445 INFO  (qtp1321832301-29535) [n:127.0.0.1:38032_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969446 INFO  (qtp1321832301-29535) [n:127.0.0.1:38032_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969446 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=foo&p=foo&t=simple-dt&e=1485325389388&s=pQLhabwWE0eUJjWJGFpOeFJ4wWY="; Domain=127.0.0.1; Expires=Rab, 25-Jan-2017 06:23:09 GMT; HttpOnly". Invalid 'expires' attribute: Rab, 25-Jan-2017 06:23:09 GMT
   [junit4]   2> 2969446 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationOtherOp
   [junit4]   2> 2969481 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenewFail
   [junit4]   2> 2969486 WARN  (qtp1321832301-29539) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 2969487 WARN  (qtp1321832301-29539) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 2969487 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969488 WARN  (qtp1824882325-29529) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 2969488 WARN  (qtp1824882325-29529) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 2969488 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969496 WARN  (qtp1321832301-29526) [n:127.0.0.1:38032_solr    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 2969496 WARN  (qtp1321832301-29526) [n:127.0.0.1:38032_solr    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 2969497 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969497 WARN  (qtp1824882325-29530) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 2969498 WARN  (qtp1824882325-29530) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 2969498 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969498 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenewFail
   [junit4]   2> 2969534 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenSolrClient
   [junit4]   2> 2969541 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969543 INFO  (qtp1321832301-29526) [n:127.0.0.1:38032_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJX2ligFZ9jIBpQsDFFtsNkQ7gArBJwy5YjYqjgmSJBDgB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969543 INFO  (qtp1321832301-29526) [n:127.0.0.1:38032_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJX2ligFZ9jIBpQsDFFtsNkQ7gArBJwy5YjYqjgmSJBDgB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969544 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenSolrClient
   [junit4]   2> 2969579 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenVerify
   [junit4]   2> 2969589 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969591 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969592 INFO  (qtp1824882325-29523) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJX3TigFZ9jIB0wwDFCivnvNIE_wV4oY8jx4U6sZgu4tsB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969592 INFO  (qtp1824882325-29523) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJX3TigFZ9jIB0wwDFCivnvNIE_wV4oY8jx4U6sZgu4tsB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969596 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969601 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2969603 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46204/solr ready
   [junit4]   2> 2969604 INFO  (qtp1824882325-29529) [n:127.0.0.1:58313_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJX3TigFZ9jIB0wwDFCivnvNIE_wV4oY8jx4U6sZgu4tsB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2969604 INFO  (qtp1824882325-29529) [n:127.0.0.1:58313_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVnSJX3TigFZ9jIB0wwDFCivnvNIE_wV4oY8jx4U6sZgu4tsB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2969607 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires=Kha, 01-Jan-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: Kha, 01-Jan-1970 00:00:00 GMT
   [junit4]   2> 2969607 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenVerify
   [junit4]   2> 2969643 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZNodePaths
   [junit4]   2> 2969654 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[2E9385B7DD1AD69A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZNodePaths
   [junit4]   2> 2969656 INFO  (jetty-launcher-4539-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@34589e44{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2969657 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=261188568
   [junit4]   2> 2969657 INFO  (jetty-launcher-4539-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@de21a94{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2969659 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=89366524
   [junit4]   2> 2969663 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.c.Overseer Overseer (id=97339925310603269-127.0.0.1:38032_solr-n_0000000000) closing
   [junit4]   2> 2969665 INFO  (OverseerStateUpdate-97339925310603269-127.0.0.1:38032_solr-n_0000000000) [n:127.0.0.1:38032_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38032_solr
   [junit4]   2> 2969667 WARN  (zkCallback-4549-thread-1-processing-n:127.0.0.1:38032_solr) [n:127.0.0.1:38032_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2969668 INFO  (jetty-launcher-4539-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2969669 ERROR (Thread[Thread-10299,5,TGRP-TestDelegationWithHadoopAuth]) [n:127.0.0.1:38032_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2969674 INFO  (zkCallback-4550-thread-1-processing-n:127.0.0.1:58313_solr) [n:127.0.0.1:58313_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2969676 INFO  (zkCallback-4550-thread-2-processing-n:127.0.0.1:58313_solr) [n:127.0.0.1:58313_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58313_solr
   [junit4]   2> 2969682 INFO  (jetty-launcher-4539-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1231c036{/solr,null,UNAVAILABLE}
   [junit4]   2> 2971177 WARN  (zkCallback-4550-thread-1-processing-n:127.0.0.1:58313_solr) [n:127.0.0.1:58313_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2971177 INFO  (jetty-launcher-4539-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2971178 ERROR (Thread[Thread-10301,5,TGRP-TestDelegationWithHadoopAuth]) [n:127.0.0.1:58313_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2971182 INFO  (jetty-launcher-4539-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4568e2ff{/solr,null,UNAVAILABLE}
   [junit4]   2> 2971185 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[2E9385B7DD1AD69A]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46204 46204
   [junit4]   2> 2976265 INFO  (Thread-10282) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46204 46204
   [junit4]   2> 2976266 WARN  (Thread-10282) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/aliases.json
   [junit4]   2> 	14	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_7
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_6
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_9
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_8
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_5
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_13
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_12
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_11
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_10
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/live_nodes
   [junit4]   2> 	14	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 2976267 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[2E9385B7DD1AD69A]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_2E9385B7DD1AD69A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=819, maxMBSortInHeap=5.643311464875187, sim=RandomSimilarity(queryNorm=false): {}, locale=ms-MY, timezone=America/Regina
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=116809616,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [ConvertedLegacyTest, ResourceLoaderTest, TestFieldCache, AtomicUpdatesTest, LeaderElectionIntegrationTest, TestRecoveryHdfs, TestJsonFacets, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerRolesTest, TestCryptoKeys, TestDynamicFieldCollectionResource, TestCollectionAPI, SSLMigrationTest, HdfsDirectoryFactoryTest, TestEmbeddedSolrServerConstructors, DistributedQueryComponentCustomSortTest, CheckHdfsIndexTest, TestNoOpRegenerator, VersionInfoTest, TestConfigSets, TestSolr4Spatial, CoreAdminCreateDiscoverTest, RestartWhileUpdatingTest, NoCacheHeaderTest, TestSolrIndexConfig, ZkSolrClientTest, JsonLoaderTest, TestLockTree, QueryParsingTest, HdfsChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, TestWriterPerf, TestStressRecovery, LoggingHandlerTest, TestSQLHandler, TestInitParams, TestCloudDeleteByQuery, RegexBoostProcessorTest, HdfsRecoveryZkTest, TestDocSet, AsyncCallRequestStatusResponseTest, CacheHeaderTest, SolrIndexSplitterTest, DistribJoinFromCollectionTest, TestBlobHandler, TestManagedResourceStorage, ShowFileRequestHandlerTest, DistributedFacetExistsSmallTest, CloneFieldUpdateProcessorFactoryTest, TestReloadAndDeleteDocs, HttpSolrCallGetCoreTest, TestBulkSchemaAPI, ChangedSchemaMergeTest, TestValueSourceCache, TestReload, TestReplicaProperties, TestQueryUtils, URLClassifyProcessorTest, CircularListTest, TestCodecSupport, ConjunctionSolrSpellCheckerTest, TermVectorComponentDistributedTest, TestDistributedMissingSort, TestManagedSchema, TestSolrConfigHandlerCloud, TestTrie, ForceLeaderTest, SharedFSAutoReplicaFailoverUtilsTest, TestCustomStream, SolrGraphiteReporterTest, BufferStoreTest, RecoveryAfterSoftCommitTest, RankQueryTest, MetricsHandlerTest, MissingSegmentRecoveryTest, OpenCloseCoreStressTest, HdfsSyncSliceTest, TolerantUpdateProcessorTest, SolrIndexMetricsTest, SuggesterTest, SolrSlf4jReporterTest, RequestHandlersTest, TestCrossCoreJoin, DistributedFacetPivotLongTailTest, DirectoryFactoryTest, TestLuceneMatchVersion, BasicFunctionalityTest, LeaderInitiatedRecoveryOnShardRestartTest, CollectionTooManyReplicasTest, RuleEngineTest, CdcrVersionReplicationTest, TestSubQueryTransformerDistrib, TestFoldingMultitermQuery, TestRandomFaceting, TestSegmentSorting, TestExceedMaxTermLength, AnalysisErrorHandlingTest, FieldMutatingUpdateProcessorTest, StatsComponentTest, AliasIntegrationTest, TestZkChroot, CollectionsAPIDistributedZkTest, TestSubQueryTransformer, TestFieldCacheSanityChecker, TestUninvertingReader, ClassificationUpdateProcessorIntegrationTest, BigEndianAscendingWordDeserializerTest, FileBasedSpellCheckerTest, TestInitQParser, CdcrReplicationHandlerTest, TestRTimerTree, HdfsNNFailoverTest, TestSolrQueryResponse, TestExactStatsCache, PreAnalyzedUpdateProcessorTest, ZkStateWriterTest, BJQParserTest, EchoParamsTest, CollectionsAPISolrJTest, ConnectionReuseTest, TestConfigSetImmutable, CursorMarkTest, OverseerTaskQueueTest, SuggestComponentTest, TestSolrCoreSnapshots, DistributedDebugComponentTest, AutoCommitTest, DirectUpdateHandlerOptimizeTest, TestSolrXml, LeaderElectionContextKeyTest, TestSolrCoreProperties, TestLMDirichletSimilarityFactory, ExternalFileFieldSortTest, TestManagedStopFilterFactory, DistanceFunctionTest, TestJavabinTupleStreamParser, TestFieldTypeCollectionResource, MetricUtilsTest, SolrMetricManagerTest, TestCollapseQParserPlugin, DeleteShardTest, SpatialHeatmapFacetsTest, TestManagedSchemaThreadSafety, SpellCheckComponentTest, TestPHPSerializedResponseWriter, UpdateParamsTest, TestCloudSchemaless, TestQueryTypes, TestIndexSearcher, TestJoin, TestSearchPerf, TestTrieFacet, LeaderFailureAfterFreshStartTest, ScriptEngineTest, ClusterStateTest, CloudMLTQParserTest, TestTolerantUpdateProcessorCloud, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SolrCoreCheckLockOnStartupTest, BadCopyFieldTest, DistanceUnitsTest, CurrencyFieldOpenExchangeTest, TestFieldCacheSortRandom, PreAnalyzedFieldTest, ClusterStateUpdateTest, ZkCLITest, TestDistributedGrouping, TestRecovery, ZkControllerTest, TestRealTimeGet, DistributedTermsComponentTest, SolrCmdDistributorTest, BadIndexSchemaTest, TestLazyCores, SoftAutoCommitTest, SolrRequestParserTest, DocValuesTest, XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestReversedWildcardFilterFactory, TestOmitPositions, DocumentBuilderTest, TermVectorComponentTest, MoreLikeThisHandlerTest, TestSolrQueryParser, SolrPluginUtilsTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, CSVRequestHandlerTest, SearchHandlerTest, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, AlternateDirectoryTest, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestSolrDeletionPolicy2, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestXIncludeConfig, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, TestFastLRUCache, DateMathParserTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, RAMDirectoryFactoryTest, TestUtils, TestDocumentBuilder, SliceStateTest, UUIDFieldTest, FileUtilsTest, DistributedMLTComponentTest, DistributedIntervalFacetingTest, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestJettySolrRunner, AssignTest, CdcrBootstrapTest, ConfigSetsAPITest, ConnectionManagerTest, CreateCollectionCleanupTest, CustomCollectionTest, DeleteNodeTest, DeleteStatusTest, DistribCursorPagingTest, DistributedQueueTest, DistributedVersionInfoTest, MigrateRouteKeyTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, ReplaceNodeTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, SolrXmlInZkTest, TestCloudInspectUtil, TestCloudPivotFacet, TestCloudRecovery, TestClusterProperties, TestConfigSetsAPI, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestDistribDocBasedVersion, TestExclusionRuleCollectionAccess, TestLeaderElectionWithEmptyReplica, TestLeaderInitiatedRecoveryThread, TestMiniSolrCloudCluster, TestRandomRequestDistribution, TestRequestStatusCollectionAPI, TestSSLRandomization, TestSizeLimitedDistributedMap, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsCollectionsAPIDistributedZkTest, HdfsTlogReplayBufferedWhileIndexingTest, HdfsWriteToMultipleCollectionsTest, TestClusterStateMutator, ZkStateReaderTest, BlobRepositoryCloudTest, CachingDirectoryFactoryTest, CoreSorterTest, TestBackupRepositoryFactory, TestConfigOverlay, TestConfigSetProperties, TestCorePropertiesReload, TestImplicitCoreProperties, TestInfoStreamLogging, TestNRTOpen, TestSolrConfigHandler, RequestLoggingTest, TestConfigReload, TestHdfsBackupRestoreCore, TestReqParamsAPI, TestRestoreCore, CoreMergeIndexesAdminHandlerTest, InfoHandlerTest, SegmentsInfoRequestHandlerTest, DistributedFacetPivotLargeTest, ReplicaListTransformerTest, TestDistributedStatsComponentCardinality, SolrJmxReporterTest, TestIntervalFaceting, TestCustomDocTransformer, TestExportWriter, TestGeoJSONResponseWriter, TestRawTransformer, TestSubQueryTransformerCrossCore, TestRestManager, TestDefaultSearchFieldResource, TestFieldCollectionResource, TestSchemaSimilarityResource, TestSolrQueryParserResource, TestUniqueKeyFieldResource, DocValuesMissingTest, SpatialRPTFieldTypeTest, TestBulkSchemaConcurrent, TestPointFields, MergeStrategyTest, TestAddFieldRealTimeGet, TestComplexPhraseQParserPlugin, TestFieldSortValues, TestFilteredDocIdSet, TestHashQParserPlugin, TestMissingGroups, TestRandomCollapseQParserPlugin, TestPKIAuthenticationPlugin, TestRuleBasedAuthorizationPlugin, TestSha256AuthenticationProvider, TestDelegationWithHadoopAuth]
   [junit4] Completed [648/680 (1!)] on J1 in 8.55s, 8 tests, 1 failure <<< FAILURES!

[...truncated 62544 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1094 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1094/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.testForwarding

Error Message:
No live SolrServers available to handle this request

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
	at __randomizedtesting.SeedInfo.seed([836C439A06BD0838:62EA2A641B8BEED1]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:416)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1344)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1095)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1037)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.create1ShardCollection(TestSolrCloudWithSecureImpersonation.java:189)
	at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.testForwarding(TestSolrCloudWithSecureImpersonation.java:346)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:     1) Thread[id=22992, name=jetty-launcher-3530-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)    2) Thread[id=22981, name=jetty-launcher-3530-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   1) Thread[id=22992, name=jetty-launcher-3530-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   2) Thread[id=22981, name=jetty-launcher-3530-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
	at __randomizedtesting.SeedInfo.seed([836C439A06BD0838]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:     1) Thread[id=26722, name=jetty-launcher-4026-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)    2) Thread[id=26718, name=jetty-launcher-4026-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   1) Thread[id=26722, name=jetty-launcher-4026-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   2) Thread[id=26718, name=jetty-launcher-4026-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
        at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
        at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
        at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
        at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
        at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
	at __randomizedtesting.SeedInfo.seed([836C439A06BD0838]:0)




Build Log:
[...truncated 12351 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_836C439A06BD0838-001/init-core-data-001
   [junit4]   2> 2650039 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[836C439A06BD0838]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2650041 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[836C439A06BD0838]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2650075 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[836C439A06BD0838]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_836C439A06BD0838-001/tempDir-001
   [junit4]   2> 2650075 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[836C439A06BD0838]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2650076 INFO  (Thread-6805) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2650076 INFO  (Thread-6805) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2650176 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[836C439A06BD0838]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:35854
   [junit4]   2> 2650183 INFO  (jetty-launcher-3530-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2650183 INFO  (jetty-launcher-3530-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2650186 INFO  (jetty-launcher-3530-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7b76b424{/solr,null,AVAILABLE}
   [junit4]   2> 2650186 INFO  (jetty-launcher-3530-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@645483f9{/solr,null,AVAILABLE}
   [junit4]   2> 2650186 INFO  (jetty-launcher-3530-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@ebe651d{HTTP/1.1,[http/1.1]}{127.0.0.1:32998}
   [junit4]   2> 2650186 INFO  (jetty-launcher-3530-thread-2) [    ] o.e.j.s.Server Started @2655347ms
   [junit4]   2> 2650186 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=32998}
   [junit4]   2> 2650187 INFO  (jetty-launcher-3530-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1bca092f{HTTP/1.1,[http/1.1]}{127.0.0.1:59932}
   [junit4]   2> 2650187 INFO  (jetty-launcher-3530-thread-1) [    ] o.e.j.s.Server Started @2655347ms
   [junit4]   2> 2650187 ERROR (jetty-launcher-3530-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-24T09:38:31.998Z
   [junit4]   2> 2650187 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59932}
   [junit4]   2> 2650188 ERROR (jetty-launcher-3530-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2650188 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-24T09:38:31.998Z
   [junit4]   2> 2650191 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2650191 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2650198 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35854/solr
   [junit4]   2> 2650198 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35854/solr
   [junit4]   2> 2650243 INFO  (jetty-launcher-3530-thread-2) [n:127.0.0.1:32998_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:32998_solr
   [junit4]   2> 2650244 INFO  (jetty-launcher-3530-thread-1) [n:127.0.0.1:59932_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59932_solr
   [junit4]   2> 2650246 INFO  (jetty-launcher-3530-thread-2) [n:127.0.0.1:32998_solr    ] o.a.s.c.Overseer Overseer (id=97337390654226438-127.0.0.1:32998_solr-n_0000000000) starting
   [junit4]   2> 2650262 INFO  (jetty-launcher-3530-thread-2) [n:127.0.0.1:32998_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32998_solr
   [junit4]   2> 2650279 INFO  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2650279 INFO  (OverseerStateUpdate-97337390654226438-127.0.0.1:32998_solr-n_0000000000) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2650304 INFO  (jetty-launcher-3530-thread-2) [n:127.0.0.1:32998_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 2650304 INFO  (jetty-launcher-3530-thread-2) [n:127.0.0.1:32998_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=solaris-vm,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 2650305 INFO  (jetty-launcher-3530-thread-2) [n:127.0.0.1:32998_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2650308 INFO  (jetty-launcher-3530-thread-2-EventThread) [n:127.0.0.1:32998_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2650329 INFO  (jetty-launcher-3530-thread-2) [n:127.0.0.1:32998_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_836C439A06BD0838-001/tempDir-001/node2/.
   [junit4]   2> 2650331 INFO  (jetty-launcher-3530-thread-1) [n:127.0.0.1:59932_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
   [junit4]   2> 2650331 INFO  (jetty-launcher-3530-thread-1) [n:127.0.0.1:59932_solr    ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=solaris-vm,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=staff, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
   [junit4]   2> 2650331 INFO  (jetty-launcher-3530-thread-1) [n:127.0.0.1:59932_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 2650337 INFO  (jetty-launcher-3530-thread-1-EventThread) [n:127.0.0.1:59932_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 2650377 INFO  (jetty-launcher-3530-thread-1) [n:127.0.0.1:59932_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_836C439A06BD0838-001/tempDir-001/node1/.
   [junit4]   2> 2651330 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
   [junit4]   2> 2651357 INFO  (qtp1902434678-22940) [n:127.0.0.1:32998_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2651357 INFO  (qtp1902434678-22940) [n:127.0.0.1:32998_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2651358 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostUsersGroup&p=anyHostUsersGroup&t=test&e=1485286713143&s=lpFq/GKyqYhs7U4dlXa/Sdo2gyI="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:33 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:33 GMT
   [junit4]   2> 2651358 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
   [junit4]   2> 2651383 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
   [junit4]   2> 2651396 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=test&e=1485286713193&s=iT+De9S7/vQSN8mTo+1Z0qFTuAI="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:33 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:33 GMT
   [junit4]   2> 2651396 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
   [junit4]   2> 2651421 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
   [junit4]   2> 2651422 INFO  (qtp1902434678-22942) [n:127.0.0.1:32998_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2651423 INFO  (qtp1902434678-22942) [n:127.0.0.1:32998_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2651423 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1485286713232&s=Pk19aaUB5Ci417QkMX8xk5IzjXk="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:33 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:33 GMT
   [junit4]   2> 2651423 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
   [junit4]   2> 2651444 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
   [junit4]   2> 2651448 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noGroups&p=noGroups&t=test&e=1485286713255&s=6a8lhtI1ZY4kpvBopH3mzV06sZ8="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:33 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:33 GMT
   [junit4]   2> 2651449 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
   [junit4]   2> 2651477 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
   [junit4]   2> 2651479 INFO  (qtp1902434678-22944) [n:127.0.0.1:32998_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 2651479 INFO  (qtp1902434678-22944) [n:127.0.0.1:32998_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2651479 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostAnyUser&p=anyHostAnyUser&t=test&e=1485286713288&s=wyQ9SqrqR7sBB9Ro0tKs4yqZwR0="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:33 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:33 GMT
   [junit4]   2> 2651479 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
   [junit4]   2> 2651507 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
   [junit4]   2> 2651512 INFO  (qtp1902434678-22945) [    ] o.a.s.s.SolrDispatchFilter Error authenticating
   [junit4]   2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:128)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:404)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:310)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2651513 WARN  (qtp1902434678-22945) [    ] o.e.j.s.ServletHandler /solr/admin/collections
   [junit4]   2> org.apache.solr.common.SolrException: Error during request authentication, 
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:410)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:310)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
   [junit4]   2> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   [junit4]   2> 	at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
   [junit4]   2> 	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
   [junit4]   2> 	at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:128)
   [junit4]   2> 	at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
   [junit4]   2> 	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:404)
   [junit4]   2> 	... 26 more
   [junit4]   2> 2651514 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
   [junit4]   2> 2651535 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
   [junit4]   2> 2651540 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noHosts&p=noHosts&t=test&e=1485286713347&s=ufz5mjLlYK/P3erOxxPONnFuDb4="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:33 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:33 GMT
   [junit4]   2> 2651541 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
   [junit4]   2> 2651562 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
   [junit4]   2> 2676598 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=wrongHost&p=wrongHost&t=test&e=1485286713372&s=4v1tpgILUq45tiAGOig3ylKRUhs="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:33 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:33 GMT
   [junit4]   2> 2676599 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
   [junit4]   2> 2676631 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
   [junit4]   2> 2676647 WARN  (qtp1902434678-22943) [    ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: invalid user name: "bar"
   [junit4]   2> 
   [junit4]   2> 2676647 WARN  (qtp1902434678-22943) [    ] o.a.h.s.UserGroupInformation No groups available for user bar
   [junit4]   2> 2676649 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bogusGroup&p=bogusGroup&t=test&e=1485286738441&s=/6BrD7IiBltdNe16ggavsoLeLfw="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:58 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:58 GMT
   [junit4]   2> 2676650 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
   [junit4]   2> 2676683 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidRemote
   [junit4]   2> 2676689 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1485286738495&s=ZXxUoBzGLy24FplbwTxZH8SXUtk="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:58 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:58 GMT
   [junit4]   2> 2676689 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidRemote
   [junit4]   2> 2676698 WARN  (zkCallback-3540-thread-3-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@232953b0 name: ZooKeeperConnection Watcher:127.0.0.1:35854/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2676698 WARN  (zkCallback-3540-thread-3-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2676699 INFO  (jetty-launcher-3530-thread-1-EventThread) [n:127.0.0.1:59932_solr    ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
   [junit4]   2> 2676703 WARN  (zkCallback-3541-thread-12-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@443889e6 name: ZooKeeperConnection Watcher:127.0.0.1:35854/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
   [junit4]   2> 2676703 WARN  (zkCallback-3541-thread-12-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 2676728 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testForwarding
   [junit4]   2> 2677624 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[836C439A06BD0838]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35854/solr ready
   [junit4]   2> 2677625 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testForwarding
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.method=testForwarding -Dtests.seed=836C439A06BD0838 -Dtests.slow=true -Dtests.locale=ar-OM -Dtests.timezone=Kwajalein -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.94s J0 | TestSolrCloudWithSecureImpersonation.testForwarding <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([836C439A06BD0838:62EA2A641B8BEED1]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:416)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1344)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1095)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1037)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
   [junit4]    > 	at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.create1ShardCollection(TestSolrCloudWithSecureImpersonation.java:189)
   [junit4]    > 	at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.testForwarding(TestSolrCloudWithSecureImpersonation.java:346)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2677659 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
   [junit4]   2> 2682809 WARN  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[836C439A06BD0838]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1485286739470&s=vHMURx4sN5XkEpT6BrH06gw+gpM="; Path=/; Domain=127.0.0.1; Expires= , 24-   -2017 19:38:59 GMT; HttpOnly". Invalid 'expires' attribute: , 24- -2017 19:38:59 GMT
   [junit4]   2> 2682810 INFO  (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[836C439A06BD0838]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
   [junit4]   2> 2682811 WARN  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@443889e6 name: ZooKeeperConnection Watcher:127.0.0.1:35854/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2682812 WARN  (OverseerStateUpdate-97337390654226438-127.0.0.1:32998_solr-n_0000000000) [n:127.0.0.1:32998_solr    ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:339)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:336)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:384)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:160)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:138)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:179)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2682812 INFO  (OverseerStateUpdate-97337390654226438-127.0.0.1:32998_solr-n_0000000000) [n:127.0.0.1:32998_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:32998_solr
   [junit4]   2> 2682812 WARN  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2682812 ERROR (OverseerExitThread) [n:127.0.0.1:32998_solr    ] o.a.s.c.Overseer could not read the data
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:286)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2682813 INFO  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.Overseer Overseer (id=97337390654226438-127.0.0.1:32998_solr-n_0000000000) closing
   [junit4]   2> 2682814 WARN  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2682814 WARN  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@232953b0 name: ZooKeeperConnection Watcher:127.0.0.1:35854/solr got event WatchedEvent state:Expired type:None path:null path: null type: None
   [junit4]   2> 2682814 WARN  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
   [junit4]   2> 2682815 WARN  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one...
   [junit4]   2> 2682817 INFO  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2682817 INFO  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2682817 INFO  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
   [junit4]   2> 2682817 INFO  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration.
   [junit4]   2> 2682818 INFO  (jetty-launcher-3530-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@ebe651d{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2682819 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1739641490
   [junit4]   2> 2682820 INFO  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 2682820 INFO  (jetty-launcher-3530-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1bca092f{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2682820 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1323076029
   [junit4]   2> 2682822 INFO  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
   [junit4]   2> 2684325 ERROR (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:305)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:389)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:305)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2684325 ERROR (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:389)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:305)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:389)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:305)
   [junit4]   2> 	... 10 more
   [junit4]   2> 
   [junit4]   2> 2684325 WARN  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2684326 INFO  (zkCallback-3540-thread-1-processing-n:127.0.0.1:59932_solr) [n:127.0.0.1:59932_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2684326 ERROR (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /security.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:305)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.addSecuritynodeWatcher(ZkStateReader.java:439)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:305)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2684326 INFO  (jetty-launcher-3530-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2684327 ERROR (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:389)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:168)
   [junit4]   2> 	at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
   [junit4]   2> 	at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:142)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /security.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:305)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:302)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.addSecuritynodeWatcher(ZkStateReader.java:439)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:426)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$1.command(ZkController.java:305)
   [junit4]   2> 	... 10 more
   [junit4]   2> 
   [junit4]   2> 2684327 WARN  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed
   [junit4]   2> 2684327 ERROR (Thread[Thread-6825,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:59932_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2684328 INFO  (zkCallback-3541-thread-9-processing-n:127.0.0.1:32998_solr) [n:127.0.0.1:32998_solr    ] o.a.s.c.c.ConnectionManager Connected:true
   [junit4]   2> 2684329 INFO  (jetty-launcher-3530-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2684329 ERROR (Thread[Thread-6822,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:32998_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2684333 INFO  (jetty-launcher-3530-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@645483f9{/solr,null,UNAVAILABLE}
   [junit4]   2> 2684337 INFO  (jetty-launcher-3530-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7b76b424{/solr,null,UNAVAILABLE}
   [junit4]   2> 2684339 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[836C439A06BD0838]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35854 35854
   [junit4]   2> 2684358 INFO  (Thread-6805) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35854 35854
   [junit4]   2> 2684360 WARN  (Thread-6805) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/clusterprops.json
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/clusterstate.json
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/collections
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 2684388 INFO  (SUITE-TestSolrCloudWithSecureImpersonation-seed#[836C439A06BD0838]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_836C439A06BD0838-001
   [junit4]   2> Jan 24, 2017 9:39:06 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> Jan 24, 2017 9:39:26 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]   2>    1) Thread[id=22992, name=jetty-launcher-3530-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2>    2) Thread[id=22981, name=jetty-launcher-3530-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2>         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2>         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2>         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2>         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2>         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> Jan 24, 2017 9:39:26 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=22992, name=jetty-launcher-3530-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2>    2) Thread[id=22981, name=jetty-launcher-3530-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]   2> 2704600 ERROR (jetty-launcher-3530-thread-2-EventThread) [n:127.0.0.1:32998_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 2704600 ERROR (jetty-launcher-3530-thread-1-EventThread) [n:127.0.0.1:59932_solr    ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]   2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]   2> 	at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]   2> 	at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]   2> 	at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]   2> 	at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]   2> 	at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 2704602 WARN  (jetty-launcher-3530-thread-1-EventThread) [n:127.0.0.1:59932_solr    ] o.a.c.ConnectionState Session expired event received
   [junit4]   2> 2704602 WARN  (jetty-launcher-3530-thread-2-EventThread) [n:127.0.0.1:32998_solr    ] o.a.c.ConnectionState Session expired event received
   [junit4]   2> 2704604 WARN  (jetty-launcher-3530-thread-1-SendThread(127.0.0.1:35854)) [n:127.0.0.1:59932_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> Jan 24, 2017 9:39:26 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> 2704604 WARN  (jetty-launcher-3530-thread-2-SendThread(127.0.0.1:35854)) [n:127.0.0.1:32998_solr    ] o.a.z.ClientCnxn Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
   [junit4]   2> java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=835, maxMBSortInHeap=6.090309748842893, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-OM, timezone=Kwajalein
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=5,free=271083576,total=530579456
   [junit4]   2> NOTE: All tests run in this JVM: [TestGroupingSearch, TestNumericTerms64, InfoHandlerTest, TestCloudPseudoReturnFields, StatelessScriptUpdateProcessorFactoryTest, PathHierarchyTokenizerFactoryTest, PeerSyncTest, AssignTest, TestSchemaResource, DocumentAnalysisRequestHandlerTest, TestRuleBasedAuthorizationPlugin, SimpleFacetsTest, OverseerModifyCollectionTest, TestMiniSolrCloudCluster, RequiredFieldsTest, TestDocBasedVersionConstraints, JSONWriterTest, MBeansHandlerTest, TestFunctionQuery, SearchHandlerTest, TestConfigSetProperties, CdcrReplicationDistributedZkTest, TestSimpleQParserPlugin, TestInfoStreamLogging, SolrCloudExampleTest, ZkControllerTest, ZkStateReaderTest, BlockDirectoryTest, UtilsToolTest, CoreAdminRequestStatusTest, TestIBSimilarityFactory, TestRemoteStreaming, HdfsBasicDistributedZk2Test, SimpleCollectionCreateDeleteTest, PreAnalyzedFieldManagedSchemaCloudTest, StressHdfsTest, TestSort, TestPointFields, DocumentBuilderTest, DOMUtilTest, TestSolrDeletionPolicy1, TestFastOutputStream, TestXIncludeConfig, MultiTermTest, TestDefaultSearchFieldResource, FileUtilsTest, TestSortByMinMaxFunction, TestSolrCloudSnapshots, TestUnifiedSolrHighlighter, TestHdfsUpdateLog, EnumFieldTest, SampleTest, TestPseudoReturnFields, TestDocTermOrds, TestJmxIntegration, TestDynamicLoading, TestSolrJ, DisMaxRequestHandlerTest, TriLevelCompositeIdRoutingTest, DistributedSpellCheckComponentTest, SolrJmxReporterTest, SyncSliceTest, HdfsRestartWhileUpdatingTest, HdfsRecoverLeaseTest, HdfsChaosMonkeyNothingIsSafeTest, SparseHLLTest, TestMiniSolrCloudClusterSSL, RecoveryZkTest, SolrGangliaReporterTest, DocValuesTest, ActionThrottleTest, TestUseDocValuesAsStored2, BaseCdcrDistributedZkTest, SpatialFilterTest, CursorPagingTest, TestFastLRUCache, OverriddenZkACLAndCredentialsProvidersTest, BlockJoinFacetSimpleTest, TestCollationFieldDocValues, RemoteQueryErrorTest, TestConfigReload, TestPhraseSuggestions, SaslZkACLProviderTest, CopyFieldTest, BlockJoinFacetDistribTest, TestAuthorizationFramework, TestFieldTypeResource, PrimUtilsTest, SolrCoreMetricManagerTest, DocValuesMultiTest, TestElisionMultitermQuery, TestLFUCache, ResponseHeaderTest, SchemaVersionSpecificBehaviorTest, TestTolerantSearch, TestWordDelimiterFilterFactory, TestSchemaVersionResource, TestHashQParserPlugin, DistributedFacetPivotSmallAdvancedTest, DistributedQueryComponentOptimizationTest, TestHashPartitioner, DateFieldTest, TestFileDictionaryLookup, DistributedQueryElevationComponentTest, TestBinaryResponseWriter, TestComplexPhraseLeadingWildcard, DateRangeFieldTest, TestManagedSchemaAPI, SharedFSAutoReplicaFailoverTest, ZkCLITest, TestConfigSetsAPI, HdfsCollectionsAPIDistributedZkTest, BooleanFieldTest, OverseerStatusTest, ClusterStateUpdateTest, TestCollationField, TestXmlQParser, DeleteLastCustomShardedReplicaTest, JavabinLoaderTest, PluginInfoTest, ConcurrentDeleteAndCreateCollectionTest, IndexSchemaRuntimeFieldTest, PeerSyncReplicationTest, TimeZoneUtilsTest, TestBM25SimilarityFactory, TestCoreContainer, TestManagedSynonymFilterFactory, DirectUpdateHandlerTest, UniqFieldsUpdateProcessorFactoryTest, TestPushWriter, SolrIndexConfigTest, DistribDocExpirationUpdateProcessorTest, TestTolerantUpdateProcessorRandomCloud, HdfsDirectoryTest, TestSweetSpotSimilarityFactory, PrimitiveFieldTypeTest, DistributedMLTComponentTest, SuggesterFSTTest, TestLeaderElectionZkExpiry, TestFieldCacheSort, TestUniqueKeyFieldResource, SOLR749Test, TestReRankQParserPlugin, TestRebalanceLeaders, TestDeleteCollectionOnDownNodes, AddSchemaFieldsUpdateProcessorFactoryTest, MoreLikeThisHandlerTest, TestLRUStatsCache, SuggestComponentContextFilterQueryTest, ConnectionManagerTest, RulesTest, ZkNodePropsTest, TestNRTOpen, CdcrUpdateLogTest, TestPerFieldSimilarity, TestExportWriter, TestAuthenticationFramework, TestScoreJoinQPNoScore, SolrMetricReporterTest, SolrTestCaseJ4Test, TestPartialUpdateDeduplication, BasicDistributedZkTest, TestSolrCLIRunExample, LeaderElectionTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, TestRandomDVFaceting, TestDistributedSearch, TestFaceting, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, TestSolr4Spatial, SpellCheckComponentTest, QueryElevationComponentTest, TestFiltering, HighlighterTest, SoftAutoCommitTest, CoreAdminHandlerTest, SuggesterWFSTTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, TestQueryUtils, StandardRequestHandlerTest, TestWriterPerf, FileBasedSpellCheckerTest, RequestHandlersTest, FastVectorHighlighterTest, LoggingHandlerTest, ReturnFieldsTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestMergePolicyConfig, TestBinaryField, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, OutputWriterTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, CollectionTooManyReplicasTest, CustomCollectionTest, DeleteStatusTest, DistributedQueueTest, DocValuesNotIndexedTest, ForceLeaderTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, MultiThreadedOCPTest, ReplicationFactorTest, RollingRestartTest, TestCloudInspectUtil, TestCloudPivotFacet, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestLocalFSCloudBackupRestore, TestRandomFlRTGCloud, TestReplicaProperties, TestRequestStatusCollectionAPI, TestSolrCloudWithDelegationTokens, TestSolrCloudWithSecureImpersonation]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=836C439A06BD0838 -Dtests.slow=true -Dtests.locale=ar-OM -Dtests.timezone=Kwajalein -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestSolrCloudWithSecureImpersonation (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 
   [junit4]    >    1) Thread[id=22992, name=jetty-launcher-3530-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEv

[...truncated too long message...]

rriddenZkACLAndCredentialsProvidersTest, BlockJoinFacetSimpleTest, TestCollationFieldDocValues, RemoteQueryErrorTest, TestConfigReload, TestPhraseSuggestions, SaslZkACLProviderTest, CopyFieldTest, BlockJoinFacetDistribTest, TestAuthorizationFramework, TestFieldTypeResource, PrimUtilsTest, SolrCoreMetricManagerTest, DocValuesMultiTest, TestElisionMultitermQuery, TestLFUCache, ResponseHeaderTest, SchemaVersionSpecificBehaviorTest, TestTolerantSearch, TestWordDelimiterFilterFactory, TestSchemaVersionResource, TestHashQParserPlugin, DistributedFacetPivotSmallAdvancedTest, DistributedQueryComponentOptimizationTest, TestHashPartitioner, DateFieldTest, TestFileDictionaryLookup, DistributedQueryElevationComponentTest, TestBinaryResponseWriter, TestComplexPhraseLeadingWildcard, DateRangeFieldTest, TestManagedSchemaAPI, SharedFSAutoReplicaFailoverTest, ZkCLITest, TestConfigSetsAPI, HdfsCollectionsAPIDistributedZkTest, BooleanFieldTest, OverseerStatusTest, ClusterStateUpdateTest, TestCollationField, TestXmlQParser, DeleteLastCustomShardedReplicaTest, JavabinLoaderTest, PluginInfoTest, ConcurrentDeleteAndCreateCollectionTest, IndexSchemaRuntimeFieldTest, PeerSyncReplicationTest, TimeZoneUtilsTest, TestBM25SimilarityFactory, TestCoreContainer, TestManagedSynonymFilterFactory, DirectUpdateHandlerTest, UniqFieldsUpdateProcessorFactoryTest, TestPushWriter, SolrIndexConfigTest, DistribDocExpirationUpdateProcessorTest, TestTolerantUpdateProcessorRandomCloud, HdfsDirectoryTest, TestSweetSpotSimilarityFactory, PrimitiveFieldTypeTest, DistributedMLTComponentTest, SuggesterFSTTest, TestLeaderElectionZkExpiry, TestFieldCacheSort, TestUniqueKeyFieldResource, SOLR749Test, TestReRankQParserPlugin, TestRebalanceLeaders, TestDeleteCollectionOnDownNodes, AddSchemaFieldsUpdateProcessorFactoryTest, MoreLikeThisHandlerTest, TestLRUStatsCache, SuggestComponentContextFilterQueryTest, ConnectionManagerTest, RulesTest, ZkNodePropsTest, TestNRTOpen, CdcrUpdateLogTest, TestPerFieldSimilarity, TestExportWriter, TestAuthenticationFramework, TestScoreJoinQPNoScore, SolrMetricReporterTest, SolrTestCaseJ4Test, TestPartialUpdateDeduplication, BasicDistributedZkTest, TestSolrCLIRunExample, LeaderElectionTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, TestRandomDVFaceting, TestDistributedSearch, TestFaceting, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, TestSolr4Spatial, SpellCheckComponentTest, QueryElevationComponentTest, TestFiltering, HighlighterTest, SoftAutoCommitTest, CoreAdminHandlerTest, SuggesterWFSTTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, TestQueryUtils, StandardRequestHandlerTest, TestWriterPerf, FileBasedSpellCheckerTest, RequestHandlersTest, FastVectorHighlighterTest, LoggingHandlerTest, ReturnFieldsTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestMergePolicyConfig, TestBinaryField, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, OutputWriterTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, CollectionTooManyReplicasTest, CustomCollectionTest, DeleteStatusTest, DistributedQueueTest, DocValuesNotIndexedTest, ForceLeaderTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, MultiThreadedOCPTest, ReplicationFactorTest, RollingRestartTest, TestCloudInspectUtil, TestCloudPivotFacet, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestLocalFSCloudBackupRestore, TestRandomFlRTGCloud, TestReplicaProperties, TestRequestStatusCollectionAPI, TestSolrCloudWithDelegationTokens, TestSolrCloudWithSecureImpersonation, TlogReplayBufferedWhileIndexingTest, HdfsNNFailoverTest, HdfsSyncSliceTest, HdfsThreadLeakTest, CachingDirectoryFactoryTest, CoreSorterTest, ExitableDirectoryReaderTest, HdfsDirectoryFactoryTest, TestHdfsBackupRestoreCore, TestReplicationHandlerBackup, TestSQLHandlerNonCloud, TestSolrConfigHandlerConcurrent, DistributedSuggestComponentTest, SuggestComponentTest, TestDistributedStatsComponentCardinality, TestMacroExpander, TestCustomDocTransformer, TestJavabinTupleStreamParser, TestChildDocTransformer, TestSubQueryTransformerDistrib, TestCloudSchemaless, TestReloadDeadlock, TestSolr4Spatial2, TestSolrCoreParser, TestSolrFieldCacheMBean, TestStressUserVersions, TestJsonFacetRefinement, TestJsonFacets, TestMinMaxOnMultiValuedField, TestOrdValues, BlockJoinFacetRandomTest, GraphQueryTest, TestScoreJoinQPScore, TestJsonRequest, SimpleMLTQParserTest, TestClassicSimilarityFactory, TestDFISimilarityFactory, TestNonDefinedSimilarityFactory, TestPerFieldSimilarityWithDefaultOverride, TestDefaultStatsCache, TestExactSharedStatsCache, BasicAuthIntegrationTest, BasicAuthStandaloneTest, PKIAuthenticationIntegrationTest, TestPKIAuthenticationPlugin, TestDelegationWithHadoopAuth, TestImpersonationWithHadoopAuth]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestImpersonationWithHadoopAuth -Dtests.seed=836C439A06BD0838 -Dtests.slow=true -Dtests.locale=th -Dtests.timezone=Europe/Moscow -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | TestImpersonationWithHadoopAuth (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 
   [junit4]    >    1) Thread[id=26722, name=jetty-launcher-4026-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]    >    2) Thread[id=26718, name=jetty-launcher-4026-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
   [junit4]    >         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
   [junit4]    >         at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
   [junit4]    >         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
   [junit4]    >         at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
   [junit4]    >         at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
   [junit4]    >         at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([836C439A06BD0838]:0)
   [junit4] Completed [641/680 (2!)] on J0 in 40.35s, 10 tests, 1 error, 1 skipped <<< FAILURES!

[...truncated 62548 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1093 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1093/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([6A0291BB27CDD2D7:9D717FE3E1257D31]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1345)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11783 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/init-core-data-001
   [junit4]   2> 1312319 INFO  (SUITE-TestReplicationHandler-seed#[6A0291BB27CDD2D7]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1312320 INFO  (SUITE-TestReplicationHandler-seed#[6A0291BB27CDD2D7]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1312322 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestDetails
   [junit4]   2> 1312323 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/collection1
   [junit4]   2> 1312326 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1312327 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3f6e7a4{/solr,null,AVAILABLE}
   [junit4]   2> 1312328 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1abe5ee2{HTTP/1.1,[http/1.1]}{127.0.0.1:39918}
   [junit4]   2> 1312329 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server Started @1315902ms
   [junit4]   2> 1312329 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=39918}
   [junit4]   2> 1312329 ERROR (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1312330 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1312330 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1312330 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1312330 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T22:00:19.960Z
   [junit4]   2> 1312331 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/solr.xml
   [junit4]   2> 1312342 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/.
   [junit4]   2> 1312342 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1312359 WARN  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1312359 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1312366 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1312368 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1312370 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/./collection1
   [junit4]   2> 1312370 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1312371 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=54.896484375, floorSegmentMB=0.8857421875, forceMergeDeletesPctAllowed=26.48503204193447, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1312390 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1312390 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1312390 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1312391 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@306559f0[collection1] main]
   [junit4]   2> 1312391 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-001/collection1/conf
   [junit4]   2> 1312391 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1312391 INFO  (coreLoadExecutor-6206-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1312391 INFO  (searcherExecutor-6207-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@306559f0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1312620 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/collection1
   [junit4]   2> 1312625 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1312627 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4bf78b8b{/solr,null,AVAILABLE}
   [junit4]   2> 1312628 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7fcf8ef6{HTTP/1.1,[http/1.1]}{127.0.0.1:33883}
   [junit4]   2> 1312628 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server Started @1316201ms
   [junit4]   2> 1312628 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=33883}
   [junit4]   2> 1312629 ERROR (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1312629 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1312629 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1312629 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1312630 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T22:00:20.260Z
   [junit4]   2> 1312630 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/solr.xml
   [junit4]   2> 1312646 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/.
   [junit4]   2> 1312647 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1312658 WARN  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1312659 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1312674 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1312676 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1312678 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/./collection1
   [junit4]   2> 1312678 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1312679 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=54.896484375, floorSegmentMB=0.8857421875, forceMergeDeletesPctAllowed=26.48503204193447, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1312692 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1312692 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1312693 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1312694 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@234b6bdd[collection1] main]
   [junit4]   2> 1312694 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-002/collection1/conf
   [junit4]   2> 1312696 INFO  (coreLoadExecutor-6216-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1312697 INFO  (searcherExecutor-6217-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@234b6bdd[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1312702 INFO  (qtp252987113-11765) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1312703 INFO  (indexFetcher-6221-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 1
   [junit4]   2> 1312703 INFO  (indexFetcher-6221-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 0
   [junit4]   2> 1312703 INFO  (indexFetcher-6221-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1312964 INFO  (qtp252987113-11767) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1312965 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1312970 INFO  (qtp252987113-11769) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1312971 INFO  (qtp2020790243-11781) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=1
   [junit4]   2> 1312972 INFO  (qtp252987113-11768) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 0
   [junit4]   2> 1312973 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1312973 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@241433a1
   [junit4]   2> 1312978 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5355c76[collection1] main]
   [junit4]   2> 1312978 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1312978 INFO  (searcherExecutor-6207-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5355c76[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4]   2> 1312979 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 5
   [junit4]   2> 1313005 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1313007 INFO  (qtp252987113-11765) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1313008 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 1313008 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1485208820603
   [junit4]   2> 1313008 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1313008 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1313012 INFO  (qtp252987113-11764) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=4
   [junit4]   2> 1313013 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 4
   [junit4]   2> 1313014 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1313015 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1313015 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MetricsDirectory(MockDirectoryWrapper(RAMDirectory@c5bc953 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6badf8d7))
   [junit4]   2> 1313016 INFO  (qtp252987113-11767) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.cfe&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313018 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313021 INFO  (qtp252987113-11769) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.cfs&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313022 INFO  (qtp252987113-11768) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313022 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1980) : 0 secs (null bytes/sec) to MetricsDirectory(MockDirectoryWrapper(RAMDirectory@c5bc953 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6badf8d7))
   [junit4]   2> 1313024 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3228941828735328]
   [junit4]   2> 1313026 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1313028 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6f14cb88[collection1] main]
   [junit4]   2> 1313028 INFO  (searcherExecutor-6217-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6f14cb88[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4]   2> 1313030 INFO  (qtp2020790243-11783) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:39918/solr/collection1/replication&command=fetchindex} status=0 QTime=26
   [junit4]   2> 1313043 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1313044 INFO  (qtp2020790243-11784) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=2
   [junit4]   2> 1313046 INFO  (qtp252987113-11765) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 1
   [junit4]   2> 1313046 INFO  (qtp252987113-11764) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1313046 INFO  (qtp252987113-11764) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@241433a1
   [junit4]   2> 1313051 INFO  (qtp252987113-11764) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1a10af99[collection1] main]
   [junit4]   2> 1313051 INFO  (qtp252987113-11764) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1313051 INFO  (searcherExecutor-6207-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1a10af99[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1) Uninverting(_1(7.0.0):c1)))}
   [junit4]   2> 1313051 INFO  (qtp252987113-11764) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 5
   [junit4]   2> 1313054 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1313055 INFO  (qtp252987113-11767) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1313055 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 3
   [junit4]   2> 1313055 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1485208820676
   [junit4]   2> 1313055 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 2
   [junit4]   2> 1313055 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1313057 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=1
   [junit4]   2> 1313057 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 7
   [junit4]   2> 1313058 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1313060 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1313060 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MetricsDirectory(MockDirectoryWrapper(RAMDirectory@43720672 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3700e6a6))
   [junit4]   2> 1313060 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.cfe because it already exists
   [junit4]   2> 1313060 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.si because it already exists
   [junit4]   2> 1313061 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.cfs because it already exists
   [junit4]   2> 1313067 INFO  (qtp252987113-11769) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.cfs&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313069 INFO  (qtp252987113-11768) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.cfe&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313070 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313071 INFO  (qtp252987113-11765) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=segments_3&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313071 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=2043) : 0 secs (null bytes/sec) to MetricsDirectory(MockDirectoryWrapper(RAMDirectory@43720672 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3700e6a6))
   [junit4]   2> 1313073 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3228941828735328]
   [junit4]   2> 1313075 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1313078 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@302bec5e[collection1] main]
   [junit4]   2> 1313078 INFO  (searcherExecutor-6217-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@302bec5e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1) Uninverting(_1(7.0.0):c1)))}
   [junit4]   2> 1313079 INFO  (qtp2020790243-11785) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:39918/solr/collection1/replication&command=fetchindex} status=0 QTime=25
   [junit4]   2> 1313082 INFO  (qtp252987113-11764) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1313082 INFO  (qtp2020790243-11786) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=2
   [junit4]   2> 1313083 INFO  (qtp252987113-11767) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 1313084 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1313084 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@241433a1
   [junit4]   2> 1313088 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2b9d62ed[collection1] main]
   [junit4]   2> 1313089 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1313089 INFO  (searcherExecutor-6207-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2b9d62ed[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1) Uninverting(_1(7.0.0):c1) Uninverting(_2(7.0.0):c1)))}
   [junit4]   2> 1313089 INFO  (qtp252987113-11766) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 5
   [junit4]   2> 1313090 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1313091 INFO  (qtp252987113-11769) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1313092 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 4
   [junit4]   2> 1313092 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's version: 1485208820714
   [junit4]   2> 1313092 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 3
   [junit4]   2> 1313092 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1313094 INFO  (qtp252987113-11768) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=4&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=2
   [junit4]   2> 1313095 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 10
   [junit4]   2> 1313096 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=32, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1313099 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1313100 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MetricsDirectory(MockDirectoryWrapper(RAMDirectory@18df262 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7e34ff45))
   [junit4]   2> 1313100 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.cfe because it already exists
   [junit4]   2> 1313100 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.si because it already exists
   [junit4]   2> 1313100 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.cfs because it already exists
   [junit4]   2> 1313100 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _1.cfs because it already exists
   [junit4]   2> 1313100 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _1.cfe because it already exists
   [junit4]   2> 1313100 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _1.si because it already exists
   [junit4]   2> 1313102 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=4&qt=/replication&file=_2.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313103 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=4&qt=/replication&file=_2.cfe&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313104 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=4&qt=/replication&file=_2.cfs&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313105 INFO  (qtp252987113-11762) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=4&qt=/replication&file=segments_4&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1313105 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=2106) : 0 secs (null bytes/sec) to MetricsDirectory(MockDirectoryWrapper(RAMDirectory@18df262 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7e34ff45))
   [junit4]   2> 1313107 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3228941828735328]
   [junit4]   2> 1313111 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1313115 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5d0bdf34[collection1] main]
   [junit4]   2> 1313115 INFO  (searcherExecutor-6217-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5d0bdf34[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1) Uninverting(_1(7.0.0):c1) Uninverting(_2(7.0.0):c1)))}
   [junit4]   2> 1313115 INFO  (qtp2020790243-11779) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:39918/solr/collection1/replication&command=fetchindex} status=0 QTime=25
   [junit4]   2> 1313117 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/collection1
   [junit4]   2> 1313121 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1313124 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5be79cc2{/solr,null,AVAILABLE}
   [junit4]   2> 1313125 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1284ad33{HTTP/1.1,[http/1.1]}{127.0.0.1:45393}
   [junit4]   2> 1313125 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server Started @1316699ms
   [junit4]   2> 1313125 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/collection1/data, hostContext=/solr, hostPort=45393}
   [junit4]   2> 1313125 ERROR (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1313126 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1313126 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1313126 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1313126 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T22:00:20.756Z
   [junit4]   2> 1313127 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/solr.xml
   [junit4]   2> 1313140 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/.
   [junit4]   2> 1313140 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1313172 WARN  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1313174 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1313186 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1313195 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1313199 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/./collection1
   [junit4]   2> 1313199 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/./collection1/data/]
   [junit4]   2> 1313200 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=54.896484375, floorSegmentMB=0.8857421875, forceMergeDeletesPctAllowed=26.48503204193447, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1313215 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1313215 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1313216 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1313217 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3959ea6[collection1] main]
   [junit4]   2> 1313217 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-003/collection1/conf
   [junit4]   2> 1313218 WARN  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1313219 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 1313219 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1313219 INFO  (coreLoadExecutor-6230-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1313220 INFO  (searcherExecutor-6231-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3959ea6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1313444 WARN  (qtp345542133-11808) [    x:collection1] o.a.s.h.ReplicationHandler Exception while invoking 'details' method for replication on master 
   [junit4]   2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39918/solr: Expected mime type application/octet-stream but got text/html. <html>
   [junit4]   2> <head>
   [junit4]   2> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
   [junit4]   2> <title>Error 404 </title>
   [junit4]   2> </head>
   [junit4]   2> <body>
   [junit4]   2> <h2>HTTP ERROR: 404</h2>
   [junit4]   2> <p>Problem accessing /solr/replication. Reason:
   [junit4]   2> <pre>    Can not find: /solr/replication</pre></p>
   [junit4]   2> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.14.v20161028</a><hr/>
   [junit4]   2> </body>
   [junit4]   2> </html>
   [junit4]   2> 
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:595)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
   [junit4]   2> 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.getDetails(IndexFetcher.java:1673)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:927)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:312)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2402)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:113)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1313445 INFO  (qtp345542133-11808) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=3
   [junit4]   2> 1313446 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1284ad33{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1313446 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=744627637
   [junit4]   2> 1313447 INFO  (coreCloseExecutor-6235-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@60d6723a
   [junit4]   2> 1313448 INFO  (coreCloseExecutor-6235-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1313449 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1313449 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5be79cc2{/solr,null,UNAVAILABLE}
   [junit4]   2> 1313450 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending doTestDetails
   [junit4]   2> 1313452 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@1abe5ee2{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1313452 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1864120484
   [junit4]   2> 1313454 INFO  (coreCloseExecutor-6237-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@114a3b55
   [junit4]   2> 1313454 INFO  (coreCloseExecutor-6237-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1313456 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1313456 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3f6e7a4{/solr,null,UNAVAILABLE}
   [junit4]   2> 1313462 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@7fcf8ef6{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1313462 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1321428855
   [junit4]   2> 1313464 INFO  (coreCloseExecutor-6239-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@457f24a1
   [junit4]   2> 1313466 INFO  (coreCloseExecutor-6239-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1313467 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1313467 INFO  (TEST-TestReplicationHandler.doTestDetails-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4bf78b8b{/solr,null,UNAVAILABLE}
   [junit4]   2> 1313474 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestStopPoll
   [junit4]   2> 1313475 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/collection1
   [junit4]   2> 1313479 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1313479 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@411371cb{/solr,null,AVAILABLE}
   [junit4]   2> 1313480 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2a553085{HTTP/1.1,[http/1.1]}{127.0.0.1:37315}
   [junit4]   2> 1313480 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server Started @1317054ms
   [junit4]   2> 1313480 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/collection1/data, hostContext=/solr, hostPort=37315}
   [junit4]   2> 1313480 ERROR (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1313480 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1313481 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1313481 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1313481 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T22:00:21.111Z
   [junit4]   2> 1313481 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/solr.xml
   [junit4]   2> 1313492 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/.
   [junit4]   2> 1313492 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1313501 WARN  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1313501 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1313523 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1313525 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1313526 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/./collection1
   [junit4]   2> 1313527 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/./collection1/data/]
   [junit4]   2> 1313527 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=54.896484375, floorSegmentMB=0.8857421875, forceMergeDeletesPctAllowed=26.48503204193447, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1313539 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1313539 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1313539 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1313540 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@18b1b33e[collection1] main]
   [junit4]   2> 1313540 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-004/collection1/conf
   [junit4]   2> 1313540 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1313541 INFO  (coreLoadExecutor-6246-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1313541 INFO  (searcherExecutor-6247-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@18b1b33e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1313871 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/collection1
   [junit4]   2> 1313873 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1313875 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3643fc4f{/solr,null,AVAILABLE}
   [junit4]   2> 1313876 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@9ffa41{HTTP/1.1,[http/1.1]}{127.0.0.1:41520}
   [junit4]   2> 1313876 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.Server Started @1317450ms
   [junit4]   2> 1313876 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/collection1/data, hostContext=/solr, hostPort=41520}
   [junit4]   2> 1313876 ERROR (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1313876 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1313876 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1313877 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1313877 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T22:00:21.507Z
   [junit4]   2> 1313877 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/solr.xml
   [junit4]   2> 1313887 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/.
   [junit4]   2> 1313887 INFO  (TEST-TestReplicationHandler.doTestStopPoll-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1313901 WARN  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1313902 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1313911 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1313912 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1313920 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/./collection1
   [junit4]   2> 1313920 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/./collection1/data/]
   [junit4]   2> 1313921 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=54.896484375, floorSegmentMB=0.8857421875, forceMergeDeletesPctAllowed=26.48503204193447, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1313932 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1313932 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1313933 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1313933 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@65435faf[collection1] main]
   [junit4]   2> 1313933 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001/solr-instance-005/collection1/conf
   [junit4]   2> 1313935 INFO  (coreLoadExecutor-6256-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1313935 INFO  (searcherExecutor-6257-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@65435faf[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1314189 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1314190 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 0
   [junit4]   2> 1314191 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 1314191 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 1314192 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 1314193 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 1314194 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 1314194 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 1314195 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 1314195 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 1314196 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 1314196 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 1314196 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 1314197 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 1314197 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 1314198 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 1314198 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 0
   [junit4]   2> 1314199 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 1314200 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 1314200 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 1314201 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 1314201 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 1314201 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 1314202 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 1314202 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 0
   [junit4]   2> 1314203 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 1314203 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 1314204 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 1314204 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 1314205 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 1314206 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 1314206 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 1314207 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 0
   [junit4]   2> 1314207 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 1314208 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 1314208 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 1314209 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 1314210 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 1314210 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 1314211 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 1314211 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 0
   [junit4]   2> 1314212 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 1314212 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 1314213 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 1314213 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 1314214 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 1314214 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 1314214 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 1314215 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 0
   [junit4]   2> 1314216 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 1314216 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 1314217 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 1314218 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 1314219 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 1314219 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 1314220 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 1314220 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 0
   [junit4]   2> 1314221 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 1314221 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 1314222 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 1314222 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 1314223 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 1314223 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 1314224 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 1314224 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 0
   [junit4]   2> 1314225 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 1314226 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 1314226 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 1314227 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 1314227 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 1314228 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 1314228 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 1314229 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 0
   [junit4]   2> 1314229 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 1314229 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 1314229 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 1314229 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 1314229 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 1314229 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 1314234 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 1314234 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 0
   [junit4]   2> 1314235 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 1314235 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 1314236 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 1314236 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 1314237 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 1314237 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 1314238 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 1314238 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 0
   [junit4]   2> 1314239 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 1314239 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 1314240 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 1314241 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 1314242 INFO  (qtp1350527993-11834) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 1314242 INFO  (qtp1350527993-11836) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 1314243 INFO  (qtp1350527993-11830) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 1314243 INFO  (qtp1350527993-11829) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 0
   [junit4]   2> 1314244 INFO  (qtp1350527993-11832) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 1314244 INFO  (qtp1350527993-11833) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 1314245 INFO  (qtp1350527993-11835) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 1314245 INFO  (

[...truncated too long message...]

rCore@6e33c744
   [junit4]   2> 1412246 INFO  (searcherExecutor-6852-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6fc9fad0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4]   2> 1412246 INFO  (Thread-3415) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1412330 INFO  (qtp754194752-12834) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 1412330 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.h.TestReplicationHandler Waited for 500ms and found 1 docs
   [junit4]   2> 1412331 INFO  (qtp701807068-12815) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1412331 INFO  (qtp701807068-12808) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1412334 INFO  (qtp701807068-12810) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1412334 INFO  (qtp754194752-12827) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=1
   [junit4]   2> 1412335 INFO  (qtp701807068-12811) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1412335 INFO  (qtp754194752-12829) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1412335 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.SolrTestCaseJ4 ###Ending doTestIndexAndConfigReplication
   [junit4]   2> 1412337 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3e425b0c{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1412337 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1126765643
   [junit4]   2> 1412337 INFO  (coreCloseExecutor-6857-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@387aad69
   [junit4]   2> 1412338 INFO  (coreCloseExecutor-6857-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1412339 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1412339 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@51aceee7{/solr,null,UNAVAILABLE}
   [junit4]   2> 1412343 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5e6c54e8{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1412343 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=969308320
   [junit4]   2> 1412344 INFO  (coreCloseExecutor-6859-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6e18d2e5
   [junit4]   2> 1412345 INFO  (coreCloseExecutor-6859-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1412346 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1412346 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[6A0291BB27CDD2D7]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7fd88b82{/solr,null,UNAVAILABLE}
   [junit4]   2> 1412350 INFO  (SUITE-TestReplicationHandler-seed#[6A0291BB27CDD2D7]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_6A0291BB27CDD2D7-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {name=BlockTreeOrds(blocksize=128), id=BlockTreeOrds(blocksize=128), newname=BlockTreeOrds(blocksize=128)}, docValues:{}, maxPointsInLeafNode=865, maxMBSortInHeap=7.0757956102178925, sim=RandomSimilarity(queryNorm=false): {}, locale=es-BO, timezone=America/North_Dakota/Center
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=113297688,total=353452032
   [junit4]   2> NOTE: All tests run in this JVM: [DeleteNodeTest, ConfigSetsAPITest, SignatureUpdateProcessorFactoryTest, SpellPossibilityIteratorTest, ChaosMonkeySafeLeaderTest, HLLUtilTest, BlockCacheTest, TestJsonFacets, TestQueryUtils, TestSolrConfigHandler, SolrIndexSplitterTest, DirectoryFactoryTest, TestZkChroot, CdcrVersionReplicationTest, TestBulkSchemaAPI, TestFieldCacheWithThreads, TestRangeQuery, TestRTimerTree, HdfsTlogReplayBufferedWhileIndexingTest, TestSuggestSpellingConverter, TestSchemaNameResource, TemplateUpdateProcessorTest, DistributedSuggestComponentTest, MinimalSchemaTest, TestLockTree, SolrGraphiteReporterTest, BasicFunctionalityTest, TestJmxMonitoredMap, SpellCheckCollatorTest, TestStressRecovery, TestQueryTypes, MetricUtilsTest, TestGeoJSONResponseWriter, ReturnFieldsTest, TestSubQueryTransformer, TestExceedMaxTermLength, TestLeaderElectionWithEmptyReplica, CollectionReloadTest, ConnectionReuseTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestMaxScoreQueryParser, TestFuzzyAnalyzedSuggestions, TestSubQueryTransformerDistrib, URLClassifyProcessorTest, CursorMarkTest, LeaderFailureAfterFreshStartTest, TestImpersonationWithHadoopAuth, DistribJoinFromCollectionTest, FileBasedSpellCheckerTest, AnalyticsMergeStrategyTest, TestManagedSchemaThreadSafety, TermVectorComponentDistributedTest, TestSolrIndexConfig, HdfsBasicDistributedZkTest, ScriptEngineTest, JvmMetricsTest, TestTestInjection, TestSmileRequest, TestOrdValues, TestIndexSearcher, TestSolr4Spatial, LukeRequestHandlerTest, OverseerTaskQueueTest, LoggingHandlerTest, BasicAuthIntegrationTest, TestSearcherReuse, GraphQueryTest, WordBreakSolrSpellCheckerTest, ResponseLogComponentTest, TestFreeTextSuggestions, ShardRoutingCustomTest, TestSolrDeletionPolicy2, TestExpandComponent, TestBlendedInfixSuggestions, SolrCLIZkUtilsTest, TestSystemIdResolver, TestManagedSchema, BadComponentTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestConfigSetsAPIExclusivity, SolrCmdDistributorTest, TestConfig, NotRequiredUniqueKeyTest, TestIndexingPerformance, BadCopyFieldTest, PreAnalyzedUpdateProcessorTest, BufferStoreTest, SuggesterTSTTest, TestTrackingShardHandlerFactory, TestSolrCoreProperties, TestDocSet, CreateCollectionCleanupTest, StatsComponentTest, SolrMetricManagerTest, LeaderElectionContextKeyTest, TestQueryWrapperFilter, OpenCloseCoreStressTest, EchoParamsTest, AnalysisErrorHandlingTest, BigEndianAscendingWordDeserializerTest, TestSolrQueryParserResource, TestDFRSimilarityFactory, CloneFieldUpdateProcessorFactoryTest, TestLMJelinekMercerSimilarityFactory, SpatialRPTFieldTypeTest, TestManagedStopFilterFactory, TestLeaderInitiatedRecoveryThread, TestFieldCacheSortRandom, TestDistributedGrouping, TestCollectionAPI, DateMathParserTest, CircularListTest, BlobRepositoryCloudTest, RequestHandlersTest, ReplicaListTransformerTest, AtomicUpdatesTest, BJQParserTest, SharedFSAutoReplicaFailoverUtilsTest, TestDistribIDF, TestReload, TestCustomStream, DefaultValueUpdateProcessorTest, SortByFunctionTest, FullSolrCloudDistribCmdsTest, TestGraphTermsQParserPlugin, UUIDFieldTest, TestQuerySenderListener, BasicDistributedZk2Test, TestFastWriter, DistributedFacetPivotSmallTest, TestImplicitCoreProperties, RegexBoostProcessorTest, ClassificationUpdateProcessorIntegrationTest, TestSha256AuthenticationProvider, TestCodecSupport, ClusterStateTest, TestUninvertingReader, DeleteShardTest, TestClassicSimilarityFactory, ManagedSchemaRoundRobinCloudTest, DirectUpdateHandlerOptimizeTest, ReplaceNodeTest, TermVectorComponentTest, HdfsChaosMonkeySafeLeaderTest, TestLuceneMatchVersion, TestRandomFaceting, SpatialHeatmapFacetsTest, DocValuesMissingTest, DistributedTermsComponentTest, SpellingQueryConverterTest, TestBackupRepositoryFactory, TestUseDocValuesAsStored, SolrIndexMetricsTest, TestPKIAuthenticationPlugin, TestCloudManagedSchema, CoreMergeIndexesAdminHandlerTest, FieldAnalysisRequestHandlerTest, TestPHPSerializedResponseWriter, TestNumericTerms32, TestEmbeddedSolrServerConstructors, TestReqParamsAPI, CachingDirectoryFactoryTest, DistributedFacetExistsSmallTest, DistributedQueryComponentCustomSortTest, MissingSegmentRecoveryTest, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, TestReplicationHandler]
   [junit4] Completed [337/680 (1!)] on J1 in 100.05s, 15 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 63516 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1092 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1092/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  org.apache.lucene.facet.TestDrillSideways.testRandom

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([22672C60F50A0F70:502B096F446AB903]:0)
	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
	at org.apache.lucene.facet.AssertingSubDocsAtOnceCollector.setScorer(AssertingSubDocsAtOnceCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.facet.DrillSidewaysScorer.score(DrillSidewaysScorer.java:86)
	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:69)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
	at org.apache.lucene.facet.DrillSideways.search(DrillSideways.java:214)
	at org.apache.lucene.facet.TestDrillSideways.testRandom(TestDrillSideways.java:796)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.facet.TestParallelDrillSideways.testRandom

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([22672C60F50A0F70:502B096F446AB903]:0)
	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
	at org.apache.lucene.facet.AssertingSubDocsAtOnceCollector.setScorer(AssertingSubDocsAtOnceCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.facet.DrillSidewaysScorer.score(DrillSidewaysScorer.java:86)
	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:69)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
	at org.apache.lucene.facet.DrillSideways.search(DrillSideways.java:214)
	at org.apache.lucene.facet.TestDrillSideways.testRandom(TestDrillSideways.java:796)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.join.TestBlockJoin.testBugCausedByRewritingTwice

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([19C321C2742C3631:E6BFF169FF13EA48]:0)
	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
	at org.apache.lucene.search.join.ToParentBlockJoinCollector$1.setScorer(ToParentBlockJoinCollector.java:190)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:182)
	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:69)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
	at org.apache.lucene.search.join.TestBlockJoin.testBugCausedByRewritingTwice(TestBlockJoin.java:406)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.join.TestBlockJoin.testMultiChildTypes

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([19C321C2742C3631:8D5422E87FA94DF2]:0)
	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
	at org.apache.lucene.search.join.ToParentBlockJoinCollector$1.setScorer(ToParentBlockJoinCollector.java:190)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:182)
	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:79)
	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:63)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
	at org.apache.lucene.search.join.TestBlockJoin.testMultiChildTypes(TestBlockJoin.java:1207)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.join.TestBlockJoin.testSimple

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([19C321C2742C3631:2170053C53DFE2E0]:0)
	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
	at org.apache.lucene.search.join.ToParentBlockJoinCollector$1.setScorer(ToParentBlockJoinCollector.java:190)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:182)
	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:79)
	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:63)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
	at org.apache.lucene.search.join.TestBlockJoin.testSimple(TestBlockJoin.java:304)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosVersion

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([619B9D838D6F1E29:9945086BF503CF7A]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:857)
	at org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosVersion(SegmentsInfoRequestHandlerTest.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=2=count(//lst[@name='segments']/lst/str[@name='version'][.='7.0.0'])
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">2</int></lst><lst name="segments"><lst name="_0"><str name="name">_0</str><int name="delCount">1</int><long name="sizeInBytes">1740</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.551Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_1"><str name="name">_1</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.555Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_2"><str name="name">_2</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.559Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_3"><str name="name">_3</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.571Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_4"><str name="name">_4</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.573Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_5"><str name="name">_5</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.575Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst></lst>
</response>

	request was:qt=/admin/segments&wt=xml
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:850)
	... 40 more




Build Log:
[...truncated 6765 lines...]
   [junit4] Suite: org.apache.lucene.facet.TestParallelDrillSideways
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestParallelDrillSideways -Dtests.method=testRandom -Dtests.seed=22672C60F50A0F70 -Dtests.slow=true -Dtests.locale=nl -Dtests.timezone=Africa/Conakry -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 2.12s J1 | TestParallelDrillSideways.testRandom <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([22672C60F50A0F70:502B096F446AB903]:0)
   [junit4]    > 	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
   [junit4]    > 	at org.apache.lucene.facet.AssertingSubDocsAtOnceCollector.setScorer(AssertingSubDocsAtOnceCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.facet.DrillSidewaysScorer.score(DrillSidewaysScorer.java:86)
   [junit4]    > 	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
   [junit4]    > 	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:69)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
   [junit4]    > 	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
   [junit4]    > 	at org.apache.lucene.facet.DrillSideways.search(DrillSideways.java:214)
   [junit4]    > 	at org.apache.lucene.facet.TestDrillSideways.testRandom(TestDrillSideways.java:796)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {foobar=IB LL-D2, field=DFR I(ne)B2, content=IB LL-D1}, locale=nl, timezone=Africa/Conakry
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=83229776,total=174063616
   [junit4]   2> NOTE: All tests run in this JVM: [TestLRUHashMap, TestParallelDrillSideways]
   [junit4] Completed [3/25 (1!)] on J1 in 3.45s, 6 tests, 1 failure <<< FAILURES!

[...truncated 46 lines...]
   [junit4] Suite: org.apache.lucene.facet.TestDrillSideways
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDrillSideways -Dtests.method=testRandom -Dtests.seed=22672C60F50A0F70 -Dtests.slow=true -Dtests.locale=ar-KW -Dtests.timezone=Asia/Harbin -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.28s J1 | TestDrillSideways.testRandom <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([22672C60F50A0F70:502B096F446AB903]:0)
   [junit4]    > 	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
   [junit4]    > 	at org.apache.lucene.facet.AssertingSubDocsAtOnceCollector.setScorer(AssertingSubDocsAtOnceCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.facet.DrillSidewaysScorer.score(DrillSidewaysScorer.java:86)
   [junit4]    > 	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
   [junit4]    > 	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:69)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
   [junit4]    > 	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
   [junit4]    > 	at org.apache.lucene.facet.DrillSideways.search(DrillSideways.java:214)
   [junit4]    > 	at org.apache.lucene.facet.TestDrillSideways.testRandom(TestDrillSideways.java:796)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {dim4=PostingsFormat(name=Memory), $full_path$=PostingsFormat(name=LuceneVarGapFixedInterval), $facets=PostingsFormat(name=Memory), dim3=PostingsFormat(name=LuceneVarGapFixedInterval), field=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), dim2=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), $payloads$=PostingsFormat(name=LuceneVarGapFixedInterval), dim1=FST50, dim0=PostingsFormat(name=Memory), id=PostingsFormat(name=LuceneVarGapFixedInterval), content=FST50}, docValues:{dim4=DocValuesFormat(name=Memory), $facets=DocValuesFormat(name=Memory), dim3=DocValuesFormat(name=Asserting), dim2=DocValuesFormat(name=Direct), dim1=DocValuesFormat(name=Lucene70), dim0=DocValuesFormat(name=Memory), id=DocValuesFormat(name=Asserting), content=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1812, maxMBSortInHeap=5.781113094024426, sim=RandomSimilarity(queryNorm=false): {foobar=ClassicSimilarity, field=DFR I(ne)L3(800.0), content=IB SPL-L3(800.0)}, locale=ar-KW, timezone=Asia/Harbin
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=132696248,total=236978176
   [junit4]   2> NOTE: All tests run in this JVM: [TestLRUHashMap, TestParallelDrillSideways, TestFacetsConfig, TestDirectoryTaxonomyWriter, TestTaxonomyFacetCounts, TestTaxonomyFacetSumValueSource, TestSearcherTaxonomyManager, TestConcurrentFacetedIndexing, TestDirectoryTaxonomyReader, TestDrillDownQuery, TestDrillSideways]
   [junit4] Completed [19/25 (2!)] on J1 in 1.38s, 6 tests, 1 failure <<< FAILURES!

[...truncated 575 lines...]
   [junit4] Suite: org.apache.lucene.search.join.TestBlockJoin
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBlockJoin -Dtests.method=testBugCausedByRewritingTwice -Dtests.seed=19C321C2742C3631 -Dtests.slow=true -Dtests.locale=ja-JP -Dtests.timezone=SST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.93s J1 | TestBlockJoin.testBugCausedByRewritingTwice <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([19C321C2742C3631:E6BFF169FF13EA48]:0)
   [junit4]    > 	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
   [junit4]    > 	at org.apache.lucene.search.join.ToParentBlockJoinCollector$1.setScorer(ToParentBlockJoinCollector.java:190)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:182)
   [junit4]    > 	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
   [junit4]    > 	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:69)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
   [junit4]    > 	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
   [junit4]    > 	at org.apache.lucene.search.join.TestBlockJoin.testBugCausedByRewritingTwice(TestBlockJoin.java:406)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBlockJoin -Dtests.method=testMultiChildTypes -Dtests.seed=19C321C2742C3631 -Dtests.slow=true -Dtests.locale=ja-JP -Dtests.timezone=SST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.04s J1 | TestBlockJoin.testMultiChildTypes <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([19C321C2742C3631:8D5422E87FA94DF2]:0)
   [junit4]    > 	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
   [junit4]    > 	at org.apache.lucene.search.join.ToParentBlockJoinCollector$1.setScorer(ToParentBlockJoinCollector.java:190)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:182)
   [junit4]    > 	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:79)
   [junit4]    > 	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:63)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
   [junit4]    > 	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
   [junit4]    > 	at org.apache.lucene.search.join.TestBlockJoin.testMultiChildTypes(TestBlockJoin.java:1207)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBlockJoin -Dtests.method=testSimple -Dtests.seed=19C321C2742C3631 -Dtests.slow=true -Dtests.locale=ja-JP -Dtests.timezone=SST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.04s J1 | TestBlockJoin.testSimple <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([19C321C2742C3631:2170053C53DFE2E0]:0)
   [junit4]    > 	at org.apache.lucene.search.AssertingScorer.getChildren(AssertingScorer.java:81)
   [junit4]    > 	at org.apache.lucene.search.join.ToParentBlockJoinCollector$1.setScorer(ToParentBlockJoinCollector.java:190)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.FilterLeafCollector.setScorer(FilterLeafCollector.java:38)
   [junit4]    > 	at org.apache.lucene.search.AssertingLeafCollector.setScorer(AssertingLeafCollector.java:43)
   [junit4]    > 	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:182)
   [junit4]    > 	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:79)
   [junit4]    > 	at org.apache.lucene.search.AssertingBulkScorer.score(AssertingBulkScorer.java:63)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
   [junit4]    > 	at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:72)
   [junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:463)
   [junit4]    > 	at org.apache.lucene.search.join.TestBlockJoin.testSimple(TestBlockJoin.java:304)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {country=DFR GL1, qualification=DFR I(n)B2, parent=DFI(Saturated), docType=DFR I(n)BZ(0.3), skill=DFR I(n)Z(0.3), childText=DFR I(ne)BZ(0.3), parentText=DFR I(ne)L1, child=IB SPL-DZ(0.3)}, locale=ja-JP, timezone=SST
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=88727728,total=239075328
   [junit4]   2> NOTE: All tests run in this JVM: [TestBlockJoin]
   [junit4] Completed [2/6 (1!)] on J1 in 6.39s, 21 tests, 3 failures <<< FAILURES!

[...truncated 3910 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.SegmentsInfoRequestHandlerTest_619B9D838D6F1E29-001/init-core-data-001
   [junit4]   2> 858483 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 858485 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 858485 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 858486 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 858526 WARN  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 858530 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 858554 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 858627 WARN  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.s.IndexSchema [null] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 858628 WARN  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 858628 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.s.IndexSchema Loaded schema test/1.4 with uniqueid field id
   [junit4]   2> 858647 WARN  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 858650 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@bac0a35
   [junit4]   2> 858650 WARN  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 858654 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@bac0a35
   [junit4]   2> 858654 WARN  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 858654 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@bac0a35
   [junit4]   2> 858656 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 858692 WARN  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 858693 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 858711 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 858779 WARN  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 858781 WARN  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 858781 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.4 with uniqueid field id
   [junit4]   2> 858795 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 858796 WARN  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 858796 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@bac0a35
   [junit4]   2> 858796 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.SegmentsInfoRequestHandlerTest_619B9D838D6F1E29-001/init-core-data-001/]
   [junit4]   2> 858797 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@bac0a35
   [junit4]   2> 858825 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 858825 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 858826 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@570c1bed[collection1] main]
   [junit4]   2> 858826 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 858830 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 858853 INFO  (coreLoadExecutor-4784-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 858854 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 858855 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 858855 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 858855 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 858855 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 858855 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 858856 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 858856 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 858856 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 858856 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 858856 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR1000]} 0 0
   [junit4]   2> 858857 INFO  (searcherExecutor-4785-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@570c1bed[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 858884 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR1001]} 0 25
   [junit4]   2> 858885 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR1002]} 0 0
   [junit4]   2> 858887 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR1003]} 0 2
   [junit4]   2> 858888 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR1004]} 0 0
   [junit4]   2> 858888 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{delete=[SOLR1000]} 0 0
   [junit4]   2> 858888 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 858888 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@60db147f
   [junit4]   2> 858899 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2e17151c[collection1] main]
   [junit4]   2> 858900 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 858901 INFO  (searcherExecutor-4785-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2e17151c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C2/1:delGen=1) Uninverting(_1(7.0.0):C2) Uninverting(_2(7.0.0):C1)))}
   [junit4]   2> 858901 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 12
   [junit4]   2> 858902 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR2000]} 0 0
   [junit4]   2> 858903 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR2001]} 0 1
   [junit4]   2> 858904 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR2002]} 0 0
   [junit4]   2> 858905 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR2003]} 0 1
   [junit4]   2> 858906 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[SOLR2004]} 0 0
   [junit4]   2> 858906 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 858906 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@60db147f
   [junit4]   2> 858911 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6835c883[collection1] main]
   [junit4]   2> 858911 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 858913 INFO  (searcherExecutor-4785-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6835c883[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C2/1:delGen=1) Uninverting(_1(7.0.0):C2) Uninverting(_2(7.0.0):C1) Uninverting(_3(7.0.0):C2) Uninverting(_4(7.0.0):C2) Uninverting(_5(7.0.0):C1)))}
   [junit4]   2> 858913 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 7
   [junit4]   2> 858917 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfos-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSegmentInfos
   [junit4]   2> 858920 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfos-seed#[619B9D838D6F1E29]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={qt=/admin/segments&wt=xml} status=0 QTime=2
   [junit4]   2> 858921 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfos-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSegmentInfos
   [junit4]   2> 858928 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosVersion-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSegmentInfosVersion
   [junit4]   2> 858931 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosVersion-seed#[619B9D838D6F1E29]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={qt=/admin/segments&wt=xml} status=0 QTime=2
   [junit4]   2> 858931 ERROR (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosVersion-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=2=count(//lst[@name='segments']/lst/str[@name='version'][.='7.0.0'])
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">2</int></lst><lst name="segments"><lst name="_0"><str name="name">_0</str><int name="delCount">1</int><long name="sizeInBytes">1740</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.551Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_1"><str name="name">_1</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.555Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_2"><str name="name">_2</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.559Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_3"><str name="name">_3</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.571Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_4"><str name="name">_4</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.573Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_5"><str name="name">_5</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.575Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst></lst>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:qt=/admin/segments&wt=xml
   [junit4]   2> 858932 ERROR (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosVersion-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 REQUEST FAILED: qt=/admin/segments&wt=xml:java.lang.RuntimeException: REQUEST FAILED: xpath=2=count(//lst[@name='segments']/lst/str[@name='version'][.='7.0.0'])
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">2</int></lst><lst name="segments"><lst name="_0"><str name="name">_0</str><int name="delCount">1</int><long name="sizeInBytes">1740</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.551Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_1"><str name="name">_1</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.555Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_2"><str name="name">_2</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.559Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_3"><str name="name">_3</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.571Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_4"><str name="name">_4</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.573Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_5"><str name="name">_5</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.575Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst></lst>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:qt=/admin/segments&wt=xml
   [junit4]   2> 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:850)
   [junit4]   2> 	at org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosVersion(SegmentsInfoRequestHandlerTest.java:70)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 858932 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosVersion-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSegmentInfosVersion
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SegmentsInfoRequestHandlerTest -Dtests.method=testSegmentInfosVersion -Dtests.seed=619B9D838D6F1E29 -Dtests.slow=true -Dtests.locale=en-AU -Dtests.timezone=America/Manaus -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.01s J1 | SegmentsInfoRequestHandlerTest.testSegmentInfosVersion <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([619B9D838D6F1E29:9945086BF503CF7A]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:857)
   [junit4]    > 	at org.apache.solr.handler.admin.SegmentsInfoRequestHandlerTest.testSegmentInfosVersion(SegmentsInfoRequestHandlerTest.java:70)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=2=count(//lst[@name='segments']/lst/str[@name='version'][.='7.0.0'])
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">2</int></lst><lst name="segments"><lst name="_0"><str name="name">_0</str><int name="delCount">1</int><long name="sizeInBytes">1740</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.551Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_1"><str name="name">_1</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.555Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_2"><str name="name">_2</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.559Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_3"><str name="name">_3</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.571Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_4"><str name="name">_4</str><int name="delCount">0</int><long name="sizeInBytes">1673</long><int name="size">2</int><date name="age">2017-01-23T10:49:16.573Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst><lst name="_5"><str name="name">_5</str><int name="delCount">0</int><long name="sizeInBytes">1632</long><int name="size">1</int><date name="age">2017-01-23T10:49:16.575Z</date><str name="source">flush</str><str name="version">7.0.0</str></lst></lst>
   [junit4]    > </response>
   [junit4]    > 	request was:qt=/admin/segments&wt=xml
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:850)
   [junit4]    > 	... 40 more
   [junit4]   2> 858938 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosData-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testSegmentInfosData
   [junit4]   2> 858941 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosData-seed#[619B9D838D6F1E29]) [    ] o.a.s.c.S.Request [collection1]  webapp=null path=null params={qt=/admin/segments&wt=xml} status=0 QTime=2
   [junit4]   2> 858941 INFO  (TEST-SegmentsInfoRequestHandlerTest.testSegmentInfosData-seed#[619B9D838D6F1E29]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testSegmentInfosData
   [junit4]   2> 858942 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 858942 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=962619950
   [junit4]   2> 858942 INFO  (coreCloseExecutor-4789-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@39202a5b
   [junit4]   2> 858977 INFO  (coreCloseExecutor-4789-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 858980 INFO  (SUITE-SegmentsInfoRequestHandlerTest-seed#[619B9D838D6F1E29]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.admin.SegmentsInfoRequestHandlerTest_619B9D838D6F1E29-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=en-AU, timezone=America/Manaus
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=217044800,total=452984832
   [junit4]   2> NOTE: All tests run in this JVM: [TestRawResponseWriter, ReplicaListTransformerTest, ConfigSetsAPITest, TestTrie, ExternalFileFieldSortTest, TestSizeLimitedDistributedMap, SortByFunctionTest, TestDFRSimilarityFactory, MetricUtilsTest, HdfsRecoveryZkTest, TestUseDocValuesAsStored, TestGraphTermsQParserPlugin, BasicDistributedZk2Test, RankQueryTest, TestClusterStateMutator, SolrIndexSplitterTest, HdfsChaosMonkeySafeLeaderTest, SolrMetricsIntegrationTest, CleanupOldIndexTest, ConvertedLegacyTest, CacheHeaderTest, TestManagedSchemaThreadSafety, TestAtomicUpdateErrorCases, SolrCLIZkUtilsTest, UpdateRequestProcessorFactoryTest, TestFieldCacheSortRandom, TestGeoJSONResponseWriter, SolrInfoMBeanTest, SpatialHeatmapFacetsTest, OpenCloseCoreStressTest, TestDynamicFieldCollectionResource, CollectionsAPISolrJTest, CoreMergeIndexesAdminHandlerTest, DistributedQueryComponentCustomSortTest, ReplaceNodeTest, TestSolrConfigHandlerCloud, SuggestComponentTest, MissingSegmentRecoveryTest, CreateCollectionCleanupTest, CloneFieldUpdateProcessorFactoryTest, BadCopyFieldTest, TestRangeQuery, SignatureUpdateProcessorFactoryTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestIndexingPerformance, TestSolrIndexConfig, TestSuggestSpellingConverter, DocValuesMissingTest, VersionInfoTest, TestCoreDiscovery, TestSolrQueryParser, DistributedDebugComponentTest, DistributedFacetExistsSmallTest, ManagedSchemaRoundRobinCloudTest, TestLMJelinekMercerSimilarityFactory, TlogReplayBufferedWhileIndexingTest, TestInitParams, BufferStoreTest, RegexBoostProcessorTest, OutOfBoxZkACLAndCredentialsProvidersTest, TermVectorComponentDistributedTest, TestSolrConfigHandler, PKIAuthenticationIntegrationTest, OverseerCollectionConfigSetProcessorTest, CoreAdminCreateDiscoverTest, FullSolrCloudDistribCmdsTest, TestConfigSetImmutable, DirectoryFactoryTest, BlockCacheTest, QueryParsingTest, TestSolrCoreSnapshots, TestAnalyzedSuggestions, SuggesterTest, SuggesterWFSTTest, TestSolrCoreParser, SolrCmdDistributorTest, RAMDirectoryFactoryTest, TestJmxMonitoredMap, TestBulkSchemaAPI, CdcrReplicationHandlerTest, DeleteInactiveReplicaTest, EchoParamsTest, LeaderElectionContextKeyTest, AsyncCallRequestStatusResponseTest, AnalysisErrorHandlingTest, TestHdfsBackupRestoreCore, TestPKIAuthenticationPlugin, TestQueryUtils, LukeRequestHandlerTest, CollectionsAPIDistributedZkTest, SpatialRPTFieldTypeTest, RestartWhileUpdatingTest, AutoCommitTest, TestClassicSimilarityFactory, TestSystemIdResolver, StatsComponentTest, BJQParserTest, JsonLoaderTest, TestPHPSerializedResponseWriter, TestSimpleTrackingShardHandler, BasicFunctionalityTest, ConnectionReuseTest, HdfsDirectoryFactoryTest, TestSearchPerf, TestFoldingMultitermQuery, OverseerRolesTest, TestCloudDeleteByQuery, SegmentsInfoRequestHandlerTest]
   [junit4] Completed [195/680 (1!)] on J1 in 0.52s, 3 tests, 1 error <<< FAILURES!

[...truncated 64003 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1091 - Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1091/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)  at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)  at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:202)  at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:137)  at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)  at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:102)  at sun.reflect.GeneratedConstructorAccessor125.newInstance(Unknown Source)  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)  at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:755)  at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:817)  at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1067)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:932)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:825)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:927)  at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:571)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
	at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)
	at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:202)
	at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:137)
	at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)
	at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:102)
	at sun.reflect.GeneratedConstructorAccessor125.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:755)
	at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:817)
	at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1067)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:932)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:825)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:927)
	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:571)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


	at __randomizedtesting.SeedInfo.seed([B47179DEFA09D67B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:289)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
	at __randomizedtesting.SeedInfo.seed([B47179DEFA09D67B:3C25460454F5BB83]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
	at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
	at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
	at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11282 lines...]
   [junit4] Suite: org.apache.solr.cloud.PeerSyncReplicationTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/init-core-data-001
   [junit4]   2> 1136832 INFO  (SUITE-PeerSyncReplicationTest-seed#[B47179DEFA09D67B]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1136833 INFO  (SUITE-PeerSyncReplicationTest-seed#[B47179DEFA09D67B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1136834 INFO  (SUITE-PeerSyncReplicationTest-seed#[B47179DEFA09D67B]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_syw/
   [junit4]   2> 1136835 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1136836 INFO  (Thread-1482) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1136836 INFO  (Thread-1482) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1136936 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.ZkTestServer start zk server on port:49759
   [junit4]   2> 1136950 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1136952 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1136954 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1136956 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1136958 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1136959 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1136961 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1136963 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1136964 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1136966 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1136968 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1137136 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/control-001/cores/collection1
   [junit4]   2> 1137137 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1137139 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@676d0e3c{/_syw,null,AVAILABLE}
   [junit4]   2> 1137139 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2f43b0ae{HTTP/1.1,[http/1.1]}{127.0.0.1:38328}
   [junit4]   2> 1137139 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server Started @1140913ms
   [junit4]   2> 1137143 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/tempDir-001/control/data, hostContext=/_syw, hostPort=38328, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/control-001/cores}
   [junit4]   2> 1137144 ERROR (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1137144 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1137144 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1137144 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1137144 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T00:21:56.092Z
   [junit4]   2> 1137148 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1137148 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/control-001/solr.xml
   [junit4]   2> 1137156 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49759/solr
   [junit4]   2> 1137184 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:38328__syw    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38328__syw
   [junit4]   2> 1137185 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:38328__syw    ] o.a.s.c.Overseer Overseer (id=97329539696885764-127.0.0.1:38328__syw-n_0000000000) starting
   [junit4]   2> 1137193 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:38328__syw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38328__syw
   [junit4]   2> 1137195 INFO  (zkCallback-1828-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1137278 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:38328__syw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/control-001/cores
   [junit4]   2> 1137278 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:38328__syw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1137283 INFO  (OverseerStateUpdate-97329539696885764-127.0.0.1:38328__syw-n_0000000000) [n:127.0.0.1:38328__syw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1137299 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1137319 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1137382 WARN  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1137383 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1137398 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1137398 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/control-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/control-001/cores/collection1/data/]
   [junit4]   2> 1137398 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fe70110
   [junit4]   2> 1137400 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=50, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7787169385760698]
   [junit4]   2> 1137409 WARN  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1137421 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1137422 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1137423 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1137423 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1137423 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=86.8525390625, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=6.286416370509997, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1137424 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@31815fb4[collection1] main]
   [junit4]   2> 1137426 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1137427 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1137427 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1137428 INFO  (coreLoadExecutor-2597-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557272635769880576
   [junit4]   2> 1137438 INFO  (coreZkRegister-2590-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1137438 INFO  (searcherExecutor-2598-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@31815fb4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1137439 INFO  (coreZkRegister-2590-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1137439 INFO  (coreZkRegister-2590-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38328/_syw/collection1/
   [junit4]   2> 1137439 INFO  (coreZkRegister-2590-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1137439 INFO  (coreZkRegister-2590-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:38328/_syw/collection1/ has no replicas
   [junit4]   2> 1137444 INFO  (coreZkRegister-2590-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38328/_syw/collection1/ shard1
   [junit4]   2> 1137470 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1137471 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49759/solr ready
   [junit4]   2> 1137471 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1137471 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1137498 INFO  (coreZkRegister-2590-thread-1-processing-n:127.0.0.1:38328__syw x:collection1 c:control_collection) [n:127.0.0.1:38328__syw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1137640 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-1-001/cores/collection1
   [junit4]   2> 1137640 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-1-001
   [junit4]   2> 1137641 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1137643 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@18e62003{/_syw,null,AVAILABLE}
   [junit4]   2> 1137643 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4e360c{HTTP/1.1,[http/1.1]}{127.0.0.1:46153}
   [junit4]   2> 1137643 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server Started @1141441ms
   [junit4]   2> 1137643 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/_syw, hostPort=46153, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-1-001/cores}
   [junit4]   2> 1137643 ERROR (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1137648 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1137648 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1137648 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1137648 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T00:21:56.596Z
   [junit4]   2> 1137651 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1137651 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-1-001/solr.xml
   [junit4]   2> 1137662 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49759/solr
   [junit4]   2> 1137675 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1137680 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46153__syw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46153__syw
   [junit4]   2> 1137684 INFO  (zkCallback-1832-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1137691 INFO  (zkCallback-1828-thread-3-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1137716 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1137766 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46153__syw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-1-001/cores
   [junit4]   2> 1137766 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46153__syw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1137783 INFO  (OverseerStateUpdate-97329539696885764-127.0.0.1:38328__syw-n_0000000000) [n:127.0.0.1:38328__syw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1137888 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1139193 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1139206 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1139317 WARN  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1139320 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1139341 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1139341 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-1-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 1139342 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fe70110
   [junit4]   2> 1139345 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=50, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7787169385760698]
   [junit4]   2> 1139352 WARN  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1139366 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1139367 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1139367 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1139367 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1139368 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=86.8525390625, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=6.286416370509997, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1139369 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4f6fd7c7[collection1] main]
   [junit4]   2> 1139370 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1139371 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1139371 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1139373 INFO  (searcherExecutor-2609-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4f6fd7c7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1139373 INFO  (coreLoadExecutor-2608-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557272637809360896
   [junit4]   2> 1139385 INFO  (coreZkRegister-2603-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1139385 INFO  (coreZkRegister-2603-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1139386 INFO  (coreZkRegister-2603-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46153/_syw/collection1/
   [junit4]   2> 1139386 INFO  (coreZkRegister-2603-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1139386 INFO  (coreZkRegister-2603-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46153/_syw/collection1/ has no replicas
   [junit4]   2> 1139387 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1139398 INFO  (coreZkRegister-2603-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46153/_syw/collection1/ shard1
   [junit4]   2> 1139536 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1139563 INFO  (coreZkRegister-2603-thread-1-processing-n:127.0.0.1:46153__syw x:collection1 c:collection1) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1139601 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-2-001/cores/collection1
   [junit4]   2> 1139601 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-2-001
   [junit4]   2> 1139602 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1139604 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@29aa83df{/_syw,null,AVAILABLE}
   [junit4]   2> 1139605 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5e90f4f5{HTTP/1.1,[http/1.1]}{127.0.0.1:35626}
   [junit4]   2> 1139605 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server Started @1143534ms
   [junit4]   2> 1139605 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/_syw, hostPort=35626, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-2-001/cores}
   [junit4]   2> 1139605 ERROR (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1139606 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1139606 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1139606 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1139606 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T00:21:58.554Z
   [junit4]   2> 1139609 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1139610 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-2-001/solr.xml
   [junit4]   2> 1139617 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49759/solr
   [junit4]   2> 1139631 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1139637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:35626__syw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35626__syw
   [junit4]   2> 1139640 INFO  (zkCallback-1832-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1139641 INFO  (zkCallback-1838-thread-1-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1139641 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1139642 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1139643 INFO  (zkCallback-1828-thread-1-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1139741 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:35626__syw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-2-001/cores
   [junit4]   2> 1139741 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:35626__syw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1139751 WARN  (OverseerStateUpdate-97329539696885764-127.0.0.1:38328__syw-n_0000000000) [n:127.0.0.1:38328__syw    ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
   [junit4]   2> 1139758 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1139767 INFO  (OverseerStateUpdate-97329539696885764-127.0.0.1:38328__syw-n_0000000000) [n:127.0.0.1:38328__syw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1139872 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1139872 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1140786 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1140798 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1140924 WARN  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1140926 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1140945 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1140945 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-2-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 1140946 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fe70110
   [junit4]   2> 1140948 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=50, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7787169385760698]
   [junit4]   2> 1140956 WARN  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1140971 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1140971 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1140973 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1140973 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1140973 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=86.8525390625, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=6.286416370509997, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1140974 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@415ade1d[collection1] main]
   [junit4]   2> 1140976 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1140976 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1140976 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1140977 INFO  (searcherExecutor-2620-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 c:collection1) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@415ade1d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1140978 INFO  (coreLoadExecutor-2619-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557272639492325376
   [junit4]   2> 1140983 INFO  (coreZkRegister-2614-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 c:collection1) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 1140984 INFO  (updateExecutor-1841-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 1140985 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 1140985 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 1140985 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 1140985 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1140985 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:46153/_syw/collection1/] and I am [http://127.0.0.1:35626/_syw/collection1/]
   [junit4]   2> 1140987 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:46153/_syw]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:35626__syw&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 1140990 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 1140991 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 1140991 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:35626__syw, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:35626/_syw","node_name":"127.0.0.1:35626__syw","state":"down"}
   [junit4]   2> 1141095 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1141096 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 1141633 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-3-001/cores/collection1
   [junit4]   2> 1141634 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-3-001
   [junit4]   2> 1141635 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1141636 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@42ee813f{/_syw,null,AVAILABLE}
   [junit4]   2> 1141637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@641bfdcf{HTTP/1.1,[http/1.1]}{127.0.0.1:46671}
   [junit4]   2> 1141637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.e.j.s.Server Started @1145614ms
   [junit4]   2> 1141637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/_syw, hostPort=46671, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-3-001/cores}
   [junit4]   2> 1141637 ERROR (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1141637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 7.0.0
   [junit4]   2> 1141637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1141637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1141637 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-23T00:22:00.585Z
   [junit4]   2> 1141642 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1141642 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-3-001/solr.xml
   [junit4]   2> 1141649 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49759/solr
   [junit4]   2> 1141666 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46671__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1141677 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46671__syw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46671__syw
   [junit4]   2> 1141682 INFO  (zkCallback-1828-thread-2-processing-n:127.0.0.1:38328__syw) [n:127.0.0.1:38328__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1141683 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1141683 INFO  (zkCallback-1832-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1141684 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1141685 INFO  (zkCallback-1851-thread-2-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1141787 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1141789 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1141894 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46671__syw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-3-001/cores
   [junit4]   2> 1141894 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [n:127.0.0.1:46671__syw    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1141926 INFO  (OverseerStateUpdate-97329539696885764-127.0.0.1:38328__syw-n_0000000000) [n:127.0.0.1:38328__syw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1141992 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:35626__syw, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:35626/_syw","node_name":"127.0.0.1:35626__syw","state":"recovering"}
   [junit4]   2> 1141992 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 1141992 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:35626__syw&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1002
   [junit4]   2> 1142039 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1142039 INFO  (zkCallback-1851-thread-2-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1142042 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1142942 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1142954 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1143028 WARN  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1143029 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1143045 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1143045 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-3-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_B47179DEFA09D67B-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 1143045 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fe70110
   [junit4]   2> 1143047 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=50, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7787169385760698]
   [junit4]   2> 1143055 WARN  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1143069 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1143069 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1143070 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1143070 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1143071 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=29, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=86.8525390625, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=6.286416370509997, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1143072 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5282789d[collection1] main]
   [junit4]   2> 1143078 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1143078 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1143078 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1143080 INFO  (searcherExecutor-2631-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 c:collection1) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5282789d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1143080 INFO  (coreLoadExecutor-2630-thread-1-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557272641696432128
   [junit4]   2> 1143086 INFO  (coreZkRegister-2625-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 c:collection1) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 1143086 INFO  (updateExecutor-1848-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 1143087 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 1143087 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 1143087 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 1143087 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1143087 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:46153/_syw/collection1/] and I am [http://127.0.0.1:46671/_syw/collection1/]
   [junit4]   2> 1143090 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:46153/_syw]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:46671__syw&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 1143093 INFO  (qtp940087244-9175) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 1143093 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1143093 INFO  (qtp940087244-9175) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 1143093 INFO  (qtp940087244-9175) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:46671__syw, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:46671/_syw","node_name":"127.0.0.1:46671__syw","state":"down"}
   [junit4]   2> 1143093 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30 for each attempt
   [junit4]   2> 1143093 INFO  (TEST-PeerSyncReplicationTest.test-seed#[B47179DEFA09D67B]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 1143193 INFO  (zkCallback-1851-thread-2-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1143193 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1143193 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1144094 INFO  (qtp940087244-9175) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:46671__syw, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:46671/_syw","node_name":"127.0.0.1:46671__syw","state":"recovering"}
   [junit4]   2> 1144094 INFO  (qtp940087244-9175) [n:127.0.0.1:46153__syw    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 1144095 INFO  (qtp940087244-9175) [n:127.0.0.1:46153__syw    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:46671__syw&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1002
   [junit4]   2> 1148993 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:46153/_syw/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 1148994 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:35626/_syw START replicas=[http://127.0.0.1:46153/_syw/collection1/] nUpdates=1000
   [junit4]   2> 1148996 INFO  (qtp940087244-9188) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 1148996 INFO  (qtp940087244-9188) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/_syw path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1148997 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 1148997 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 1148997 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1148997 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1148998 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1148998 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 1148998 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 1148998 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 1148998 INFO  (recoveryExecutor-1842-thread-1-processing-n:127.0.0.1:35626__syw x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:35626__syw c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 1149002 INFO  (zkCallback-1838-thread-2-processing-n:127.0.0.1:46153__syw) [n:127.0.0.1:46153__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1149003 INFO  (zkCallback-1851-thread-2-processing-n:127.0.0.1:46671__syw) [n:127.0.0.1:46671__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1149003 INFO  (zkCallback-1844-thread-1-processing-n:127.0.0.1:35626__syw) [n:127.0.0.1:35626__syw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 1151096 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:46153/_syw/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 1151096 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:46671/_syw START replicas=[http://127.0.0.1:46153/_syw/collection1/] nUpdates=1000
   [junit4]   2> 1151099 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 1151099 INFO  (qtp940087244-9180) [n:127.0.0.1:46153__syw c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/_syw path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1151100 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 1151100 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 1151100 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1151100 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1151101 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1151101 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:46671__syw c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 1151101 INFO  (recoveryExecutor-1849-thread-1-processing-n:127.0.0.1:46671__syw x:collection1 s:shard1 c:collection1 r:cor

[...truncated too long message...]

Total(FSNamesystem.java:5729)
   [junit4]   2> 	... 54 more
   [junit4]   2> 3048954 INFO  (SUITE-HdfsRecoveryZkTest-seed#[B47179DEFA09D67B]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:57323 57323
   [junit4]   2> 3048954 INFO  (SUITE-HdfsRecoveryZkTest-seed#[B47179DEFA09D67B]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=FSTOrd50, _version_=PostingsFormat(name=LuceneVarGapFixedInterval), a_t=PostingsFormat(name=MockRandom), a_i=FSTOrd50, id=FSTOrd50}, docValues:{}, maxPointsInLeafNode=671, maxMBSortInHeap=5.541671270952973, sim=RandomSimilarity(queryNorm=false): {}, locale=en-PH, timezone=America/Blanc-Sablon
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=2,free=62179744,total=485662720
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexingPerformance, TestLMJelinekMercerSimilarityFactory, VersionInfoTest, ShardSplitTest, HdfsWriteToMultipleCollectionsTest, TestReloadAndDeleteDocs, TestOnReconnectListenerSupport, DefaultValueUpdateProcessorTest, TestGraphTermsQParserPlugin, ConnectionReuseTest, TestDynamicFieldCollectionResource, BlockCacheTest, TestDistributedGrouping, FieldAnalysisRequestHandlerTest, TestSmileRequest, TestTrie, TestRandomRequestDistribution, TestSolrCloudWithKerberosAlt, CloudMLTQParserTest, AnalysisErrorHandlingTest, SolrCoreCheckLockOnStartupTest, TestExclusionRuleCollectionAccess, AutoCommitTest, TestValueSourceCache, WordBreakSolrSpellCheckerTest, NoCacheHeaderTest, RegexBoostProcessorTest, TestSolr4Spatial, TestSha256AuthenticationProvider, CloneFieldUpdateProcessorFactoryTest, DistanceUnitsTest, RestartWhileUpdatingTest, TestSolrConfigHandler, AnalyticsMergeStrategyTest, TestExceedMaxTermLength, TestSubQueryTransformerDistrib, CdcrBootstrapTest, TestLeaderElectionWithEmptyReplica, TestXmlQParserPlugin, SolrCmdDistributorTest, DeleteShardTest, BasicDistributedZk2Test, TestImpersonationWithHadoopAuth, SolrIndexMetricsTest, TestCustomDocTransformer, BasicAuthIntegrationTest, SortByFunctionTest, SpatialHeatmapFacetsTest, TestFieldCache, TestSolrQueryParserResource, TestUtils, BitVectorTest, BigEndianAscendingWordDeserializerTest, ChangedSchemaMergeTest, TestOrdValues, MissingSegmentRecoveryTest, TestRangeQuery, FullHLLTest, TestSegmentSorting, TestIntervalFaceting, TestReqParamsAPI, TestLuceneMatchVersion, SuggesterTest, SolrXmlInZkTest, RuleEngineTest, TestSQLHandler, SuggesterWFSTTest, TestZkChroot, TestCrossCoreJoin, ReplicaListTransformerTest, TestQueryTypes, ClassificationUpdateProcessorIntegrationTest, TestTrieFacet, CachingDirectoryFactoryTest, ShowFileRequestHandlerTest, TestFieldSortValues, NotRequiredUniqueKeyTest, SolrSlf4jReporterTest, LukeRequestHandlerTest, TestFieldCacheWithThreads, TestSchemaSimilarityResource, HighlighterMaxOffsetTest, CoreAdminHandlerTest, ClusterStateTest, TestSuggestSpellingConverter, TestSolrQueryParser, DistanceFunctionTest, TestFuzzyAnalyzedSuggestions, CursorMarkTest, TestJavabinTupleStreamParser, OverseerTaskQueueTest, TestCoreDiscovery, DistributedFacetPivotSmallTest, ScriptEngineTest, TestFieldCollectionResource, ResponseLogComponentTest, SegmentsInfoRequestHandlerTest, SuggestComponentTest, TestBulkSchemaAPI, DistribJoinFromCollectionTest, TestSearchPerf, HdfsLockFactoryTest, TestCustomStream, TestNoOpRegenerator, TestHdfsBackupRestoreCore, ExternalFileFieldSortTest, ReturnFieldsTest, TestNumericTerms32, CollectionReloadTest, TestConfigSetImmutable, SolrMetricsIntegrationTest, ConfigSetsAPITest, TestRealTimeGet, SolrCLIZkUtilsTest, TestSolrQueryResponse, CollectionsAPIDistributedZkTest, TestCopyFieldCollectionResource, ReplaceNodeTest, OpenCloseCoreStressTest, TestSizeLimitedDistributedMap, TestComplexPhraseQParserPlugin, StatsComponentTest, TestExpandComponent, SSLMigrationTest, TestFastWriter, RequestHandlersTest, SpellingQueryConverterTest, TestPKIAuthenticationPlugin, TestSolrCoreParser, TestCloudSchemaless, TestBlobHandler, TestCryptoKeys, AliasIntegrationTest, TestFieldCacheSanityChecker, TlogReplayBufferedWhileIndexingTest, SpellCheckCollatorTest, BlockJoinFacetRandomTest, TestRecovery, CheckHdfsIndexTest, TestSlowCompositeReaderWrapper, DeleteNodeTest, MetricsHandlerTest, CollectionTooManyReplicasTest, TestSolrCoreSnapshots, TestRTimerTree, TestDocSet, BadIndexSchemaTest, SolrInfoMBeanTest, TestFieldTypeCollectionResource, TestDFRSimilarityFactory, JsonLoaderTest, TestIndexSearcher, TestTrackingShardHandlerFactory, SpellPossibilityIteratorTest, MinimalSchemaTest, TestCollapseQParserPlugin, TestBackupRepositoryFactory, TestWriterPerf, RecoveryAfterSoftCommitTest, TestLMDirichletSimilarityFactory, DistribCursorPagingTest, TestSolrIndexConfig, LeaderElectionIntegrationTest, TestOmitPositions, TestComponentsName, DataDrivenBlockJoinTest, TestStressCloudBlindAtomicUpdates, BasicZkTest, RecoveryZkTest, TestReplicationHandler, ZkSolrClientTest, TestRandomDVFaceting, TestDistributedSearch, TestJoin, HardAutoCommitTest, DistributedTermsComponentTest, ConvertedLegacyTest, TestFiltering, TestFunctionQuery, TestBadConfig, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, TestFoldingMultitermQuery, DocValuesTest, DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, TermVectorComponentTest, RequiredFieldsTest, LoggingHandlerTest, IndexSchemaTest, UpdateRequestProcessorFactoryTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, SearchHandlerTest, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, UpdateParamsTest, AlternateDirectoryTest, SolrIndexConfigTest, BadComponentTest, TestSolrDeletionPolicy2, SampleTest, TestBinaryField, NumericFieldsTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, DirectSolrConnectionTest, TestCodecSupport, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestBM25SimilarityFactory, TestJmxMonitoredMap, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, RAMDirectoryFactoryTest, TestSolrJ, TestDocumentBuilder, SliceStateTest, UUIDFieldTest, FileUtilsTest, CircularListTest, TestRTGBase, DistributedIntervalFacetingTest, TestDistributedMissingSort, TestSimpleTrackingShardHandler, CdcrReplicationHandlerTest, CleanupOldIndexTest, CollectionsAPIAsyncDistributedZkTest, CreateCollectionCleanupTest, DistributedVersionInfoTest, HttpPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, OverseerCollectionConfigSetProcessorTest, OverseerRolesTest, OverseerStatusTest, RemoteQueryErrorTest, RollingRestartTest, TestAuthenticationFramework, TestCloudRecovery, TestConfigSetsAPI, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestRequestStatusCollectionAPI, TestSolrCloudWithSecureImpersonation, TestTolerantUpdateProcessorRandomCloud, TriLevelCompositeIdRoutingTest, HdfsBasicDistributedZk2Test, HdfsChaosMonkeyNothingIsSafeTest, HdfsCollectionsAPIDistributedZkTest, HdfsRecoveryZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsRecoveryZkTest -Dtests.seed=B47179DEFA09D67B -Dtests.slow=true -Dtests.locale=en-PH -Dtests.timezone=America/Blanc-Sablon -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | HdfsRecoveryZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
   [junit4]    > 	at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)
   [junit4]    > 	at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:202)
   [junit4]    > 	at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:137)
   [junit4]    > 	at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)
   [junit4]    > 	at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:102)
   [junit4]    > 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
   [junit4]    > 	at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:755)
   [junit4]    > 	at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:817)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1067)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:932)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:825)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:927)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:571)
   [junit4]    > 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B47179DEFA09D67B]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:289)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [534/680 (2!)] on J1 in 64.58s, 1 test, 1 failure <<< FAILURES!

[...truncated 62904 lines...]


[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1090 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1090/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
timeout waiting to see all nodes active

Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
	at __randomizedtesting.SeedInfo.seed([9070C66A508CF174:1824F9B0FE709C8C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.PeerSyncReplicationTest.waitTillNodesActive(PeerSyncReplicationTest.java:326)
	at org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:277)
	at org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:259)
	at org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12284 lines...]
   [junit4] Suite: org.apache.solr.cloud.PeerSyncReplicationTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/init-core-data-001
   [junit4]   2> 2620928 INFO  (SUITE-PeerSyncReplicationTest-seed#[9070C66A508CF174]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 2620930 INFO  (SUITE-PeerSyncReplicationTest-seed#[9070C66A508CF174]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2620930 INFO  (SUITE-PeerSyncReplicationTest-seed#[9070C66A508CF174]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /rcpl/k
   [junit4]   2> 2620934 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2620934 INFO  (Thread-6141) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2620934 INFO  (Thread-6141) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2621034 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.ZkTestServer start zk server on port:48387
   [junit4]   2> 2621052 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2621056 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2621059 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2621061 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2621063 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2621066 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2621069 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2621071 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2621074 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2621077 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2621080 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2621230 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/control-001/cores/collection1
   [junit4]   2> 2621231 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2621233 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@339ca386{/rcpl/k,null,AVAILABLE}
   [junit4]   2> 2621234 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@330bd700{HTTP/1.1,[http/1.1]}{127.0.0.1:38838}
   [junit4]   2> 2621234 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server Started @2626041ms
   [junit4]   2> 2621234 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/tempDir-001/control/data, hostContext=/rcpl/k, hostPort=38838, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/control-001/cores}
   [junit4]   2> 2621234 ERROR (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2621234 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2621234 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2621234 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2621235 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-22T12:28:28.667Z
   [junit4]   2> 2621240 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2621240 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/control-001/solr.xml
   [junit4]   2> 2621248 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48387/solr
   [junit4]   2> 2621287 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38838_rcpl%2Fk
   [junit4]   2> 2621289 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.Overseer Overseer (id=97326734270791684-127.0.0.1:38838_rcpl%2Fk-n_0000000000) starting
   [junit4]   2> 2621301 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38838_rcpl%2Fk
   [junit4]   2> 2621304 INFO  (OverseerStateUpdate-97326734270791684-127.0.0.1:38838_rcpl%2Fk-n_0000000000) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2621446 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/control-001/cores
   [junit4]   2> 2621446 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2621452 INFO  (OverseerStateUpdate-97326734270791684-127.0.0.1:38838_rcpl%2Fk-n_0000000000) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2622468 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2622484 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2622595 WARN  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2622599 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2622635 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2622635 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/control-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/control-001/cores/collection1/data/]
   [junit4]   2> 2622635 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@556c5fad
   [junit4]   2> 2622638 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=66.572265625, floorSegmentMB=1.0556640625, forceMergeDeletesPctAllowed=20.05078628936581, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3936373592860306
   [junit4]   2> 2622648 WARN  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2622660 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2622660 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2622663 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2622663 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2622663 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2f68a1cf
   [junit4]   2> 2622664 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2864296c[collection1] main]
   [junit4]   2> 2622666 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2622667 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2622667 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2622668 INFO  (searcherExecutor-8594-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2864296c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2622668 INFO  (coreLoadExecutor-8593-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557227750136217600
   [junit4]   2> 2622679 INFO  (coreZkRegister-8586-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2622680 INFO  (coreZkRegister-8586-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2622680 INFO  (coreZkRegister-8586-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38838/rcpl/k/collection1/
   [junit4]   2> 2622680 INFO  (coreZkRegister-8586-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2622680 INFO  (coreZkRegister-8586-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:38838/rcpl/k/collection1/ has no replicas
   [junit4]   2> 2622687 INFO  (coreZkRegister-8586-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38838/rcpl/k/collection1/ shard1
   [junit4]   2> 2622842 INFO  (coreZkRegister-8586-thread-1-processing-n:127.0.0.1:38838_rcpl%2Fk x:collection1 c:control_collection) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2623077 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2623079 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48387/solr ready
   [junit4]   2> 2623079 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2623227 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-1-001/cores/collection1
   [junit4]   2> 2623227 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-1-001
   [junit4]   2> 2623228 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2623231 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f9f2c7b{/rcpl/k,null,AVAILABLE}
   [junit4]   2> 2623231 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3afb700e{HTTP/1.1,[http/1.1]}{127.0.0.1:44084}
   [junit4]   2> 2623231 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server Started @2628039ms
   [junit4]   2> 2623232 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/rcpl/k, hostPort=44084, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-1-001/cores}
   [junit4]   2> 2623232 ERROR (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2623232 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2623232 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2623232 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2623232 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-22T12:28:30.664Z
   [junit4]   2> 2623236 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2623236 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-1-001/solr.xml
   [junit4]   2> 2623263 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48387/solr
   [junit4]   2> 2623279 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2623287 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44084_rcpl%2Fk
   [junit4]   2> 2623290 INFO  (zkCallback-3561-thread-3-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2623290 INFO  (zkCallback-3565-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2623292 INFO  (zkCallback-3570-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2623402 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-1-001/cores
   [junit4]   2> 2623402 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2623406 INFO  (OverseerStateUpdate-97326734270791684-127.0.0.1:38838_rcpl%2Fk-n_0000000000) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2624435 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2624449 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2624580 WARN  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2624583 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2624603 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2624604 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-1-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 2624604 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@556c5fad
   [junit4]   2> 2624607 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=66.572265625, floorSegmentMB=1.0556640625, forceMergeDeletesPctAllowed=20.05078628936581, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3936373592860306
   [junit4]   2> 2624620 WARN  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2624640 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2624640 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2624642 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2624642 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2624643 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@abd9b4c
   [junit4]   2> 2624644 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@14ff9d3c[collection1] main]
   [junit4]   2> 2624646 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2624647 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2624647 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2624648 INFO  (searcherExecutor-8605-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@14ff9d3c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2624648 INFO  (coreLoadExecutor-8604-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557227752212398080
   [junit4]   2> 2624660 INFO  (coreZkRegister-8599-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2624661 INFO  (coreZkRegister-8599-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2624661 INFO  (coreZkRegister-8599-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44084/rcpl/k/collection1/
   [junit4]   2> 2624661 INFO  (coreZkRegister-8599-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2624662 INFO  (coreZkRegister-8599-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:44084/rcpl/k/collection1/ has no replicas
   [junit4]   2> 2624668 INFO  (coreZkRegister-8599-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44084/rcpl/k/collection1/ shard1
   [junit4]   2> 2624821 INFO  (coreZkRegister-8599-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2625102 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-2-001/cores/collection1
   [junit4]   2> 2625102 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-2-001
   [junit4]   2> 2625103 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2625106 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b126b86{/rcpl/k,null,AVAILABLE}
   [junit4]   2> 2625106 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2522ff4c{HTTP/1.1,[http/1.1]}{127.0.0.1:33030}
   [junit4]   2> 2625106 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server Started @2629914ms
   [junit4]   2> 2625106 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/rcpl/k, hostPort=33030, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-2-001/cores}
   [junit4]   2> 2625106 ERROR (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2625107 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2625107 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2625107 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2625107 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-22T12:28:32.539Z
   [junit4]   2> 2625112 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2625112 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-2-001/solr.xml
   [junit4]   2> 2625117 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48387/solr
   [junit4]   2> 2625133 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:33030_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 2625141 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:33030_rcpl%2Fk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33030_rcpl%2Fk
   [junit4]   2> 2625143 INFO  (zkCallback-3565-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2625143 INFO  (zkCallback-3561-thread-3-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2625143 INFO  (zkCallback-3570-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2625143 INFO  (zkCallback-3576-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 2625258 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:33030_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-2-001/cores
   [junit4]   2> 2625258 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:33030_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2625261 INFO  (OverseerStateUpdate-97326734270791684-127.0.0.1:38838_rcpl%2Fk-n_0000000000) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2626277 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2626290 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2626398 WARN  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2626400 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2626419 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2626420 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-2-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 2626420 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@556c5fad
   [junit4]   2> 2626422 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=66.572265625, floorSegmentMB=1.0556640625, forceMergeDeletesPctAllowed=20.05078628936581, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3936373592860306
   [junit4]   2> 2626432 WARN  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2626445 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2626445 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2626447 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2626447 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2626447 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@16c68c3b
   [junit4]   2> 2626448 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@62175f53[collection1] main]
   [junit4]   2> 2626449 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2626450 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2626450 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2626451 INFO  (searcherExecutor-8616-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@62175f53[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2626452 INFO  (coreLoadExecutor-8615-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557227754104029184
   [junit4]   2> 2626457 INFO  (coreZkRegister-8610-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 2626458 INFO  (updateExecutor-3573-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 2626459 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 2626459 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 2626459 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 2626459 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 2626459 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:44084/rcpl/k/collection1/] and I am [http://127.0.0.1:33030/rcpl/k/collection1/]
   [junit4]   2> 2626462 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:44084/rcpl/k]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:33030_rcpl%252Fk&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 2626464 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 2626465 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 2626465 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:33030_rcpl%2Fk, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:33030/rcpl/k","node_name":"127.0.0.1:33030_rcpl%2Fk","state":"down"}
   [junit4]   2> 2627027 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-3-001/cores/collection1
   [junit4]   2> 2627028 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-3-001
   [junit4]   2> 2627029 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 2627030 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e798942{/rcpl/k,null,AVAILABLE}
   [junit4]   2> 2627031 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@68aaae26{HTTP/1.1,[http/1.1]}{127.0.0.1:47058}
   [junit4]   2> 2627031 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.e.j.s.Server Started @2631838ms
   [junit4]   2> 2627031 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/rcpl/k, hostPort=47058, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-3-001/cores}
   [junit4]   2> 2627031 ERROR (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 2627031 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 2627031 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 2627031 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 2627031 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-01-22T12:28:34.463Z
   [junit4]   2> 2627035 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2627035 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-3-001/solr.xml
   [junit4]   2> 2627042 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48387/solr
   [junit4]   2> 2627057 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:47058_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 2627064 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:47058_rcpl%2Fk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47058_rcpl%2Fk
   [junit4]   2> 2627066 INFO  (zkCallback-3561-thread-2-processing-n:127.0.0.1:38838_rcpl%2Fk) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2627066 INFO  (zkCallback-3565-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2627067 INFO  (zkCallback-3570-thread-1-processing-n:127.0.0.1:44084_rcpl%2Fk) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2627067 INFO  (zkCallback-3576-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk) [n:127.0.0.1:33030_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2627068 INFO  (zkCallback-3583-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 2627109 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:47058_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-3-001/cores
   [junit4]   2> 2627109 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [n:127.0.0.1:47058_rcpl%2Fk    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 2627113 INFO  (OverseerStateUpdate-97326734270791684-127.0.0.1:38838_rcpl%2Fk-n_0000000000) [n:127.0.0.1:38838_rcpl%2Fk    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2627467 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:33030_rcpl%2Fk, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:33030/rcpl/k","node_name":"127.0.0.1:33030_rcpl%2Fk","state":"recovering"}
   [junit4]   2> 2627467 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 2627467 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:33030_rcpl%252Fk&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1003
   [junit4]   2> 2628129 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 2628150 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2628254 WARN  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 2628255 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 2628273 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2628273 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-3-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.PeerSyncReplicationTest_9070C66A508CF174-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 2628273 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@556c5fad
   [junit4]   2> 2628275 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=66.572265625, floorSegmentMB=1.0556640625, forceMergeDeletesPctAllowed=20.05078628936581, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3936373592860306
   [junit4]   2> 2628284 WARN  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 2628298 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2628298 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=1000 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2628299 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2628299 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2628300 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@28216065
   [junit4]   2> 2628301 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1ccdf6c8[collection1] main]
   [junit4]   2> 2628326 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2628329 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2628334 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2628339 INFO  (searcherExecutor-8627-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1ccdf6c8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2628341 INFO  (coreLoadExecutor-8626-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk) [n:127.0.0.1:47058_rcpl%2Fk c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1557227756084789248
   [junit4]   2> 2628355 INFO  (coreZkRegister-8621-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 c:collection1) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 2628355 INFO  (updateExecutor-3580-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 2628356 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 2628356 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 2628356 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 2628356 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 2628356 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:44084/rcpl/k/collection1/] and I am [http://127.0.0.1:47058/rcpl/k/collection1/]
   [junit4]   2> 2628361 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:44084/rcpl/k]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:47058_rcpl%252Fk&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 2628363 INFO  (qtp184447031-22629) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 2628363 INFO  (qtp184447031-22629) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 2628363 INFO  (qtp184447031-22629) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:47058_rcpl%2Fk, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:47058/rcpl/k","node_name":"127.0.0.1:47058_rcpl%2Fk","state":"down"}
   [junit4]   2> 2628663 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 2628663 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30 for each attempt
   [junit4]   2> 2628663 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 2629371 INFO  (qtp184447031-22629) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:47058_rcpl%2Fk, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:47058/rcpl/k","node_name":"127.0.0.1:47058_rcpl%2Fk","state":"recovering"}
   [junit4]   2> 2629371 INFO  (qtp184447031-22629) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 2629371 INFO  (qtp184447031-22629) [n:127.0.0.1:44084_rcpl%2Fk    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:47058_rcpl%252Fk&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1008
   [junit4]   2> 2634470 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:44084/rcpl/k/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 2634470 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:33030/rcpl/k START replicas=[http://127.0.0.1:44084/rcpl/k/collection1/] nUpdates=1000
   [junit4]   2> 2634472 INFO  (qtp184447031-22626) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2634472 INFO  (qtp184447031-22626) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/rcpl/k path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2634473 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2634473 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 2634473 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2634473 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2634474 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2634474 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 2634474 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 2634474 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 2634474 INFO  (recoveryExecutor-3574-thread-1-processing-n:127.0.0.1:33030_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 2636373 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:44084/rcpl/k/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 2636374 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:47058/rcpl/k START replicas=[http://127.0.0.1:44084/rcpl/k/collection1/] nUpdates=1000
   [junit4]   2> 2636376 INFO  (qtp184447031-22627) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2636376 INFO  (qtp184447031-22627) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/rcpl/k path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 2636377 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 2636377 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 2636377 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2636377 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2636378 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2636378 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 2636378 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 2636378 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 2636378 INFO  (recoveryExecutor-3581-thread-1-processing-n:127.0.0.1:47058_rcpl%2Fk x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 2636691 INFO  (TEST-PeerSyncReplicationTest.test-seed#[9070C66A508CF174]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
   [junit4]   2> 2636692 INFO  (qtp2028176514-22589) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2636692 INFO  (qtp2028176514-22589) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2636693 INFO  (qtp2028176514-22589) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2636693 INFO  (qtp2028176514-22589) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
   [junit4]   2> 2636696 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2636696 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2636697 INFO  (qtp1542813611-22651) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2636697 INFO  (qtp1542813611-22651) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2636698 INFO  (qtp1542813611-22651) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2636698 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2636698 INFO  (qtp1542813611-22651) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
   [junit4]   2> 2636698 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
   [junit4]   2> 2636701 INFO  (qtp103098067-22688) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2636701 INFO  (qtp103098067-22688) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 2636701 INFO  (qtp103098067-22688) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 2636701 INFO  (qtp103098067-22688) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
   [junit4]   2> 2636701 INFO  (qtp184447031-22630) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 7
   [junit4]   2> 2636703 INFO  (qtp184447031-22626) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/rcpl/k path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2636704 INFO  (qtp1542813611-22656) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/rcpl/k path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2636705 INFO  (qtp103098067-22690) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/rcpl/k path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 2638708 INFO  (qtp2028176514-22587) [n:127.0.0.1:38838_rcpl%2Fk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1557227766954328064)} 0 1
   [junit4]   2> 2638713 INFO  (qtp103098067-22691) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&_version_=-1557227766956425216&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{deleteByQuery=*:* (-1557227766956425216)} 0 2
   [junit4]   2> 2638713 INFO  (qtp1542813611-22654) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&_version_=-1557227766956425216&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{deleteByQuery=*:* (-1557227766956425216)} 0 2
   [junit4]   2> 2638714 INFO  (qtp184447031-22627) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1557227766956425216)} 0 5
   [junit4]   2> 2638717 INFO  (qtp1542813611-22657) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[0 (1557227766962716672)]} 0 1
   [junit4]   2> 2638718 INFO  (qtp103098067-22692) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[0 (1557227766962716672)]} 0 1
   [junit4]   2> 2638718 INFO  (qtp184447031-22629) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={wt=javabin&version=2}{add=[0 (1557227766962716672)]} 0 2
   [junit4]   2> 2638719 INFO  (qtp1542813611-22658) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[1 (1557227766966910976)]} 0 0
   [junit4]   2> 2638720 INFO  (qtp103098067-22693) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[1 (1557227766966910976)]} 0 0
   [junit4]   2> 2638720 INFO  (qtp184447031-22623) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={wt=javabin&version=2}{add=[1 (1557227766966910976)]} 0 1
   [junit4]   2> 2638722 INFO  (qtp1542813611-22652) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[2 (1557227766969008128)]} 0 0
   [junit4]   2> 2638722 INFO  (qtp103098067-22686) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[2 (1557227766969008128)]} 0 0
   [junit4]   2> 2638722 INFO  (qtp184447031-22625) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={wt=javabin&version=2}{add=[2 (1557227766969008128)]} 0 1
   [junit4]   2> 2638724 INFO  (qtp1542813611-22651) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[3 (1557227766971105280)]} 0 0
   [junit4]   2> 2638724 INFO  (qtp103098067-22688) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[3 (1557227766971105280)]} 0 0
   [junit4]   2> 2638724 INFO  (qtp184447031-22630) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={wt=javabin&version=2}{add=[3 (1557227766971105280)]} 0 1
   [junit4]   2> 2638726 INFO  (qtp1542813611-22655) [n:127.0.0.1:33030_rcpl%2Fk c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[4 (1557227766973202432)]} 0 0
   [junit4]   2> 2638727 INFO  (qtp103098067-22689) [n:127.0.0.1:47058_rcpl%2Fk c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/rcpl/k path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44084/rcpl/k/collection1/&wt=javabin&version=2}{add=[4 (1557227766973202432)]} 0 0
   [junit4]   2> 2638727 INFO  (qtp184447031-22628) [n:127.0.0.1:44084_rcpl%2Fk c:collection1 

[...truncated too long message...]

search.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=11855, maxDocsPerChunk=786, blockSize=7), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=11855, blockSize=7)), sim=RandomSimilarity(queryNorm=true): {}, locale=es-CU, timezone=America/Tijuana
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=303464776,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrDeletionPolicy1, TestConfigOverlay, BooleanFieldTest, TestXIncludeConfig, TestDFISimilarityFactory, BlockJoinFacetSimpleTest, JavabinLoaderTest, SampleTest, TestMinMaxOnMultiValuedField, TestHighFrequencyDictionaryFactory, TestCharFilters, TestChildDocTransformer, MigrateRouteKeyTest, EnumFieldTest, OverseerModifyCollectionTest, PrimitiveFieldTypeTest, TestSchemalessBufferedUpdates, TestDownShardTolerantSearch, TestClusterProperties, UnloadDistributedZkTest, ClusterStateUpdateTest, TestSweetSpotSimilarityFactory, DistributedMLTComponentTest, TestShortCircuitedRequests, TestMissingGroups, TestNRTOpen, TestRequestForwarding, TestDynamicFieldResource, TestDistributedStatsComponentCardinality, CollectionStateFormat2Test, SolrCoreMetricManagerTest, TestFieldTypeResource, TestShardHandlerFactory, DistributedQueueTest, TestRuleBasedAuthorizationPlugin, HdfsCollectionsAPIDistributedZkTest, TestRemoteStreaming, TestStressUserVersions, TestReplicationHandlerBackup, StressHdfsTest, RollingRestartTest, TestSubQueryTransformerCrossCore, DistributedQueryComponentOptimizationTest, TestSolrCloudWithHadoopAuthPlugin, TestWordDelimiterFilterFactory, TestElisionMultitermQuery, TestHdfsCloudBackupRestore, SearchHandlerTest, TestDefaultSearchFieldResource, PingRequestHandlerTest, TestFieldCacheReopen, TestDistributedSearch, TestReplicationHandler, DOMUtilTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestRTGBase, TestFunctionQuery, AddBlockUpdateTest, TestSQLHandlerNonCloud, RequiredFieldsTest, TestSortByMinMaxFunction, ParsingFieldUpdateProcessorsTest, TestHdfsUpdateLog, TestPivotHelperCode, NumericFieldsTest, TestStressLiveNodes, TestPartialUpdateDeduplication, RecoveryZkTest, SimpleFacetsTest, TestCloudInspectUtil, TestSolr4Spatial2, TestQuerySenderNoQuery, TestComplexPhraseLeadingWildcard, TestRestoreCore, DocumentBuilderTest, TestPerFieldSimilarity, TestLFUCache, DeleteReplicaTest, TestReRankQParserPlugin, TestSSLRandomization, TestJsonRequest, TestFieldCacheVsDocValues, UUIDUpdateProcessorFallbackTest, TestRandomCollapseQParserPlugin, TestManagedSynonymFilterFactory, TestManagedResource, TestBulkSchemaConcurrent, TestHashPartitioner, UtilsToolTest, TestExtendedDismaxParser, SynonymTokenizerTest, CoreSorterTest, DateFieldTest, TestPseudoReturnFields, TestUpdate, CurrencyFieldXmlFileTest, TestPostingsSolrHighlighter, TestSolrCloudSnapshots, TestPerFieldSimilarityWithDefaultOverride, TestOverriddenPrefixQueryForCustomFieldType, HdfsChaosMonkeyNothingIsSafeTest, TimeZoneUtilsTest, TestStressReorder, ExplicitHLLTest, MultiThreadedOCPTest, XmlUpdateRequestHandlerTest, TestJettySolrRunner, TestSolrFieldCacheMBean, HighlighterTest, HdfsRecoverLeaseTest, TermsComponentTest, TestSimpleQParserPlugin, TestAnalyzeInfixSuggestions, TestIBSimilarityFactory, HdfsRestartWhileUpdatingTest, CollectionsAPIAsyncDistributedZkTest, HdfsDirectoryTest, PluginInfoTest, BlockDirectoryTest, HdfsBasicDistributedZk2Test, TestMiniSolrCloudClusterSSL, SolrPluginUtilsTest, TestSurroundQueryParser, IndexBasedSpellCheckerTest, TestTolerantUpdateProcessorRandomCloud, PeerSyncWithIndexFingerprintCachingTest, SystemInfoHandlerTest, TestUnifiedSolrHighlighter, TestJmxIntegration, ActionThrottleTest, OverriddenZkACLAndCredentialsProvidersTest, TestCloudPseudoReturnFields, TestPhraseSuggestions, TestBinaryResponseWriter, TestMiniSolrCloudCluster, AnalyticsQueryTest, TestHashQParserPlugin, CdcrRequestHandlerTest, TestInfoStreamLogging, SharedFSAutoReplicaFailoverTest, SOLR749Test, DistributedSpellCheckComponentTest, FileUtilsTest, TestConfigSetProperties, TestFileDictionaryLookup, TestRandomDVFaceting, BigEndianAscendingWordSerializerTest, RequestLoggingTest, TestLocalFSCloudBackupRestore, ZkCLITest, HdfsThreadLeakTest, ExitableDirectoryReaderTest, ClassificationUpdateProcessorTest, DirectUpdateHandlerTest, DocValuesNotIndexedTest, TestLeaderElectionZkExpiry, TestDocBasedVersionConstraints, PathHierarchyTokenizerFactoryTest, DistributedFacetPivotLargeTest, TestGroupingSearch, BasicAuthStandaloneTest, HdfsUnloadDistributedZkTest, CdcrReplicationDistributedZkTest, BasicDistributedZkTest, BasicDistributedZk2Test, SyncSliceTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, LeaderElectionTest, ZkSolrClientTest, ShardRoutingCustomTest, TestFaceting, TermVectorComponentDistributedTest, ZkControllerTest, TestReload, HardAutoCommitTest, TestCoreContainer, SolrCoreTest, PeerSyncTest, BasicFunctionalityTest, TestLazyCores, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, DocValuesTest, DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, MoreLikeThisHandlerTest, FastVectorHighlighterTest, IndexSchemaRuntimeFieldTest, TestCSVResponseWriter, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, HighlighterConfigTest, TestSolrIndexConfig, SolrIndexConfigTest, TestSolrDeletionPolicy2, MultiTermTest, TestBinaryField, TestFuzzyAnalyzedSuggestions, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TestFastOutputStream, ScriptEngineTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, CdcrReplicationHandlerTest, CdcrVersionReplicationTest, CleanupOldIndexTest, CollectionTooManyReplicasTest, ConnectionManagerTest, CustomCollectionTest, DeleteStatusTest, DistributedVersionInfoTest, LeaderFailoverAfterPartitionTest, LeaderFailureAfterFreshStartTest, LeaderInitiatedRecoveryOnCommitTest, PeerSyncReplicationTest]
   [junit4] Completed [500/680 (1!)] on J0 in 210.85s, 1 test, 1 failure <<< FAILURES!

[...truncated 43219 lines...]
  [javadoc] Generating Javadoc
  [javadoc] Javadoc execution
  [javadoc] Loading source files for package org.apache.lucene...
  [javadoc] Loading source files for package org.apache.lucene.analysis...
  [javadoc] Loading source files for package org.apache.lucene.analysis.standard...
  [javadoc] Loading source files for package org.apache.lucene.analysis.tokenattributes...
  [javadoc] Loading source files for package org.apache.lucene.codecs...
  [javadoc] Loading source files for package org.apache.lucene.codecs.blocktree...
  [javadoc] Loading source files for package org.apache.lucene.codecs.compressing...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene50...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene60...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene62...
  [javadoc] Loading source files for package org.apache.lucene.codecs.lucene70...
  [javadoc] Loading source files for package org.apache.lucene.codecs.perfield...
  [javadoc] Loading source files for package org.apache.lucene.document...
  [javadoc] Loading source files for package org.apache.lucene.geo...
  [javadoc] Loading source files for package org.apache.lucene.index...
  [javadoc] Loading source files for package org.apache.lucene.search...
  [javadoc] Loading source files for package org.apache.lucene.search.similarities...
  [javadoc] Loading source files for package org.apache.lucene.search.spans...
  [javadoc] Loading source files for package org.apache.lucene.store...
  [javadoc] Loading source files for package org.apache.lucene.util...
  [javadoc] Loading source files for package org.apache.lucene.util.automaton...
  [javadoc] Loading source files for package org.apache.lucene.util.bkd...
  [javadoc] Loading source files for package org.apache.lucene.util.fst...
  [javadoc] Loading source files for package org.apache.lucene.util.graph...
  [javadoc] Loading source files for package org.apache.lucene.util.mutable...
  [javadoc] Loading source files for package org.apache.lucene.util.packed...
  [javadoc] Constructing Javadoc information...
  [javadoc] javadoc: error - Argument for -bottom contains JavaScript.
  [javadoc] Use --allow-script-in-comments to allow use of JavaScript.
  [javadoc] 1 error

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:775: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/build.xml:138: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:746: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/core/build.xml:54: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:2101: Javadoc returned 1

Total time: 91 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any