You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/11/20 15:57:01 UTC
[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_144) - Build #
20957 - Still Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20957/
Java: 32bit/jdk1.8.0_144 -server -XX:+UseSerialGC
1 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 1) Thread[id=15658, name=jetty-launcher-1525-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41) at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244) at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44) at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61) at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
1) Thread[id=15658, name=jetty-launcher-1525-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
at __randomizedtesting.SeedInfo.seed([58BB826C014095CF]:0)
Build Log:
[...truncated 13420 lines...]
[junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_58BB826C014095CF-001/init-core-data-001
[junit4] 2> 2625995 WARN (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
[junit4] 2> 2625995 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 2625996 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 2625997 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 2626011 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_58BB826C014095CF-001/tempDir-001
[junit4] 2> 2626011 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2626012 INFO (Thread-1988) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2626012 INFO (Thread-1988) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2626014 ERROR (Thread-1988) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2626112 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:43675
[junit4] 2> 2626128 INFO (jetty-launcher-1525-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 2626128 INFO (jetty-launcher-1525-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 2626144 INFO (jetty-launcher-1525-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@fc9151{/solr,null,AVAILABLE}
[junit4] 2> 2626144 INFO (jetty-launcher-1525-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16c0e9b{/solr,null,AVAILABLE}
[junit4] 2> 2626151 INFO (jetty-launcher-1525-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@bed794{SSL,[ssl, http/1.1]}{127.0.0.1:42065}
[junit4] 2> 2626151 INFO (jetty-launcher-1525-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@36551e{SSL,[ssl, http/1.1]}{127.0.0.1:40621}
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-1) [ ] o.e.j.s.Server Started @2628309ms
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-2) [ ] o.e.j.s.Server Started @2628309ms
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42065}
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40621}
[junit4] 2> 2626152 ERROR (jetty-launcher-1525-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2626152 ERROR (jetty-launcher-1525-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T15:37:19.368Z
[junit4] 2> 2626152 INFO (jetty-launcher-1525-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T15:37:19.368Z
[junit4] 2> 2626154 INFO (jetty-launcher-1525-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2626154 INFO (jetty-launcher-1525-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2626160 INFO (jetty-launcher-1525-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43675/solr
[junit4] 2> 2626160 INFO (jetty-launcher-1525-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43675/solr
[junit4] 2> 2626192 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2626192 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2626192 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40621_solr
[junit4] 2> 2626192 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.Overseer Overseer (id=99037494590046214-127.0.0.1:40621_solr-n_0000000000) starting
[junit4] 2> 2626192 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42065_solr
[junit4] 2> 2626193 INFO (zkCallback-1536-thread-1-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2626193 INFO (zkCallback-1537-thread-1-processing-n:127.0.0.1:40621_solr) [n:127.0.0.1:40621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2626196 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40621_solr
[junit4] 2> 2626200 INFO (zkCallback-1536-thread-1-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2626202 INFO (zkCallback-1537-thread-1-processing-n:127.0.0.1:40621_solr) [n:127.0.0.1:40621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2626232 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 2626232 WARN (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 2626232 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 2626232 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 2626236 INFO (jetty-launcher-1525-thread-1-EventThread) [n:127.0.0.1:42065_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 2626271 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 2626271 WARN (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 2626272 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 2626275 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 2626279 WARN (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 2626281 INFO (jetty-launcher-1525-thread-2-EventThread) [n:127.0.0.1:40621_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 2626300 WARN (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 2626330 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42065.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@8d860c
[junit4] 2> 2626331 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40621.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@8d860c
[junit4] 2> 2626336 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42065.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@8d860c
[junit4] 2> 2626336 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42065.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@8d860c
[junit4] 2> 2626336 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40621.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@8d860c
[junit4] 2> 2626336 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40621.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@8d860c
[junit4] 2> 2626337 INFO (jetty-launcher-1525-thread-1) [n:127.0.0.1:42065_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_58BB826C014095CF-001/tempDir-001/node1/.
[junit4] 2> 2626337 INFO (jetty-launcher-1525-thread-2) [n:127.0.0.1:40621_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_58BB826C014095CF-001/tempDir-001/node2/.
[junit4] 2> 2626383 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
[junit4] 2> 2626413 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noHosts&p=noHosts&t=test&e=1511228239621&s=CnydG1SuzWyiO5oqnp+5Txdr40Y="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:19 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:19 GMT
[junit4] 2> 2626413 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigHosts-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
[junit4] 2> 2626424 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
[junit4] 2> 2626425 INFO (qtp29202853-15607) [n:127.0.0.1:40621_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2626425 INFO (qtp29202853-15607) [n:127.0.0.1:40621_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2626425 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1511228239640&s=/bWcb/7cm5pLgJdisFH2ZnJL81w="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:19 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:19 GMT
[junit4] 2> 2626425 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateHost-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
[junit4] 2> 2626436 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
[junit4] 2> 2636462 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1511228239652&s=GLKc19cwehd0W3Z5+nGNxD1CTzw="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:19 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:19 GMT
[junit4] 2> 2636463 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
[junit4] 2> 2636475 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
[junit4] 2> 2651489 WARN (jetty-launcher-1525-thread-2-SendThread(127.0.0.1:43675)) [n:127.0.0.1:40621_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15029ms for sessionid 0x15fda139c7c0008
[junit4] 2> 2651490 WARN (jetty-launcher-1525-thread-1-SendThread(127.0.0.1:43675)) [n:127.0.0.1:42065_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15030ms for sessionid 0x15fda139c7c0007
[junit4] 2> 2651490 WARN (jetty-launcher-1525-thread-1-SendThread(127.0.0.1:43675)) [n:127.0.0.1:42065_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15030ms for sessionid 0x15fda139c7c0005
[junit4] 2> 2651490 WARN (jetty-launcher-1525-thread-2-SendThread(127.0.0.1:43675)) [n:127.0.0.1:40621_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15030ms for sessionid 0x15fda139c7c0006
[junit4] 2> 2651492 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15fda139c7c0008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2651492 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15fda139c7c0007, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2651492 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15fda139c7c0005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2651492 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15fda139c7c0006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2651493 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=wrongHost&p=wrongHost&t=test&e=1511228249691&s=Mg7hc7wQx/ucB07s/kxvpV7lAcI="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:29 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:29 GMT
[junit4] 2> 2651493 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyWrongHost-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
[junit4] 2> 2651507 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidRemote
[junit4] 2> 2651512 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=localHostAnyGroup&p=localHostAnyGroup&t=test&e=1511228264723&s=r2XxmdeL04aoxhj69KPIUCVE6l0="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:44 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:44 GMT
[junit4] 2> 2651512 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidRemote-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidRemote
[junit4] 2> 2651527 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
[junit4] 2> 2651535 WARN (qtp29202853-15607) [n:127.0.0.1:40621_solr ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: ‘bar’: no such user
[junit4] 2> id: ‘bar’: no such user
[junit4] 2>
[junit4] 2> 2651536 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bogusGroup&p=bogusGroup&t=test&e=1511228264743&s=wvCDneqXzBTzLyakmLGLC34aXRM="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:44 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:44 GMT
[junit4] 2> 2651537 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidGroup-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
[junit4] 2> 2651580 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
[junit4] 2> 2651581 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=test&e=1511228264796&s=AXjgwL18IsmxuA9yV2uFW3n/v4c="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:44 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:44 GMT
[junit4] 2> 2651582 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyInvalidProxyUser-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
[junit4] 2> 2651591 INFO (jetty-launcher-1525-thread-1-EventThread) [n:127.0.0.1:42065_solr ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
[junit4] 2> 2651594 WARN (zkCallback-1536-thread-1-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@139f277 name: ZooKeeperConnection Watcher:127.0.0.1:43675/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 2651594 WARN (zkCallback-1536-thread-1-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 2651594 WARN (zkCallback-1537-thread-3-processing-n:127.0.0.1:40621_solr) [n:127.0.0.1:40621_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@11b4de8 name: ZooKeeperConnection Watcher:127.0.0.1:43675/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 2651595 WARN (zkCallback-1537-thread-3-processing-n:127.0.0.1:40621_solr) [n:127.0.0.1:40621_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 2651607 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
[junit4] 2> 2651607 INFO (qtp29202853-15613) [n:127.0.0.1:40621_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2651607 INFO (qtp29202853-15613) [n:127.0.0.1:40621_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2651608 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostAnyUser&p=anyHostAnyUser&t=test&e=1511228264823&s=4VhSy3DK6exo3IIPYu9NH9MkfZI="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:44 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:44 GMT
[junit4] 2> 2651608 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateAnyHostAnyUser-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
[junit4] 2> 2651619 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
[junit4] 2> 2651620 INFO (qtp29202853-15615) [ ] o.a.s.s.SolrDispatchFilter Error authenticating
[junit4] 2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
[junit4] 2> at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2651620 WARN (qtp29202853-15615) [ ] o.e.j.s.ServletHandler /solr/admin/collections
[junit4] 2> org.apache.solr.common.SolrException: Error during request authentication,
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler$HttpParamAuthenticationHandler.authenticate(HttpParamDelegationTokenPlugin.java:213)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:134)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamToRequestFilter.doFilter(HttpParamDelegationTokenPlugin.java:245)
[junit4] 2> at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:243)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> ... 29 more
[junit4] 2> 2651621 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNullProxyUser-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
[junit4] 2> 2651635 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
[junit4] 2> 2651637 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=noGroups&p=noGroups&t=test&e=1511228264852&s=tKqyBrp83L8EkxBxLrK1BO0LG6w="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:44 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:44 GMT
[junit4] 2> 2651637 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
[junit4] 2> 2651665 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
[junit4] 2> 2651676 INFO (qtp29202853-15612) [n:127.0.0.1:40621_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2651676 INFO (qtp29202853-15612) [n:127.0.0.1:40621_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2651677 WARN (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=anyHostUsersGroup&p=anyHostUsersGroup&t=test&e=1511228264881&s=3uXP18gP6D+1TxpWwz/Y2wvZplo="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:44 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:44 GMT
[junit4] 2> 2651677 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyValidateGroup-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
[junit4] 2> 2651693 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testForwarding
[junit4] 2> 2651857 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15fda139c7c000a, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2651858 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2651859 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43675/solr ready
[junit4] 2> 2651860 INFO (qtp29202853-15615) [n:127.0.0.1:40621_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2653232 INFO (OverseerThreadFactory-4619-thread-1-processing-n:127.0.0.1:40621_solr) [n:127.0.0.1:40621_solr ] o.a.s.c.CreateCollectionCmd Create collection forwardingCollection
[junit4] 2> 2653379 INFO (jetty-launcher-1525-thread-2-EventThread) [n:127.0.0.1:40621_solr ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
[junit4] 2> 2653390 INFO (OverseerStateUpdate-99037494590046214-127.0.0.1:40621_solr-n_0000000000) [n:127.0.0.1:40621_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"forwardingCollection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"forwardingCollection_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:42065/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 2653633 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 2653634 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2653746 INFO (zkCallback-1536-thread-5-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2653746 INFO (zkCallback-1536-thread-4-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2654678 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 2654741 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema [forwardingCollection_shard1_replica_n1] Schema name=test
[junit4] 2> 2654869 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 2654903 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'forwardingCollection_shard1_replica_n1' using configuration from collection forwardingCollection, trusted=true
[junit4] 2> 2654903 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42065.solr.core.forwardingCollection.shard1.replica_n1' (registry 'solr.core.forwardingCollection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@8d860c
[junit4] 2> 2654904 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2654904 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [[forwardingCollection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_58BB826C014095CF-001/tempDir-001/node1/forwardingCollection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_58BB826C014095CF-001/tempDir-001/node1/./forwardingCollection_shard1_replica_n1/data/]
[junit4] 2> 2654907 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2654994 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2654994 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2654996 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2654996 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2654999 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=86.8369140625, floorSegmentMB=0.7841796875, forceMergeDeletesPctAllowed=12.361665703792095, segmentsPerTier=48.0, maxCFSSegmentSizeMB=1.78515625, noCFSRatio=0.0
[junit4] 2> 2654999 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1b98998[forwardingCollection_shard1_replica_n1] main]
[junit4] 2> 2655001 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
[junit4] 2> 2655027 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4] 2> 2655104 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2655104 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
[junit4] 2> 2655104 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1584599943945912320
[junit4] 2> 2655109 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
[junit4] 2> 2655109 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
[junit4] 2> 2655109 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
[junit4] 2> 2655109 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
[junit4] 2> 2655109 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
[junit4] 2> 2655109 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
[junit4] 2> 2655109 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
[junit4] 2> 2655110 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
[junit4] 2> 2655111 INFO (searcherExecutor-4624-thread-1-processing-n:127.0.0.1:42065_solr x:forwardingCollection_shard1_replica_n1 s:shard1 c:forwardingCollection r:core_node2) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] Registered new searcher Searcher@1b98998[forwardingCollection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2655115 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2655115 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2655115 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42065/solr/forwardingCollection_shard1_replica_n1/
[junit4] 2> 2655115 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2655115 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:42065/solr/forwardingCollection_shard1_replica_n1/ has no replicas
[junit4] 2> 2655115 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2655117 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42065/solr/forwardingCollection_shard1_replica_n1/ shard1
[junit4] 2> 2655219 INFO (zkCallback-1536-thread-4-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2655219 INFO (zkCallback-1536-thread-5-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2655221 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2655223 INFO (qtp23679846-15614) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf1&newCollection=true&name=forwardingCollection_shard1_replica_n1&action=CREATE&numShards=1&collection=forwardingCollection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1589
[junit4] 2> 2655224 WARN (httpShardExecutor-4611-thread-1-processing-n:127.0.0.1:40621_solr [https:////127.0.0.1:42065//solr] https:////127.0.0.1:42065//solr) [n:127.0.0.1:40621_solr ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=$&p=$&t=test&e=1511228266849&s=VXbErZjVFAFAcPDFmm5PH6FZGRo="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:46 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:46 GMT
[junit4] 2> 2655226 INFO (qtp29202853-15615) [n:127.0.0.1:40621_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 2655233 INFO (OverseerCollectionConfigSetProcessor-99037494590046214-127.0.0.1:40621_solr-n_0000000000) [n:127.0.0.1:40621_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2655325 INFO (zkCallback-1536-thread-5-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2655326 INFO (zkCallback-1536-thread-4-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/forwardingCollection/state.json] for collection [forwardingCollection] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2656227 INFO (qtp29202853-15615) [n:127.0.0.1:40621_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=1&collection.configName=conf1&maxShardsPerNode=1&name=forwardingCollection&nrtReplicas=1&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=4367
[junit4] 2> 2656227 WARN (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=user&p=user&t=test&e=1511228265076&s=5kQWKtZBh70nfhYFFD0M6QJ6jF8="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:45 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:45 GMT
[junit4] 2> 2656227 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: forwardingCollection failOnTimeout:true timeout (sec):100
[junit4] 2> 2656228 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: forwardingCollection
[junit4] 2> 2656236 INFO (qtp29202853-15609) [n:127.0.0.1:40621_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2656240 INFO (qtp23679846-15605) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 2656240 WARN (qtp29202853-15609) [n:127.0.0.1:40621_solr ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=$&p=$&t=test&e=1511228269455&s=BLjRXMThUwL8C/wtTAHX/osz+pg="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:49 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:49 GMT
[junit4] 2> 2656240 WARN (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=user&p=user&t=test&e=1511228269452&s=Y7nE4xYELQNAIFY3YtuQi4qiC6w="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:49 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:49 GMT
[junit4] 2> 2656240 WARN (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=$&p=$&t=test&e=1511228269455&s=BLjRXMThUwL8C/wtTAHX/osz+pg="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:49 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:49 GMT
[junit4] 2> 2656244 INFO (qtp23679846-15602) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.S.Request [forwardingCollection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 2656244 WARN (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=user&p=user&t=test&e=1511228269459&s=qHcrJi1HeGZt8GjzXJaS7X5s18o="; Path=/; Domain=127.0.0.1; Expires=mar, 21-nov-2017 01:37:49 GMT; Secure; HttpOnly". Invalid 'expires' attribute: mar, 21-nov-2017 01:37:49 GMT
[junit4] 2> 2656245 INFO (TEST-TestSolrCloudWithSecureImpersonation.testForwarding-seed#[58BB826C014095CF]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testForwarding
[junit4] 2> 2656274 INFO (jetty-closer-1526-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@bed794{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 2656274 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=20141995
[junit4] 2> 2656275 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2656275 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@c1195: rootName = solr_42065, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@10933a0
[junit4] 2> 2656281 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2656281 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7fb029: rootName = solr_42065, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@26b02e
[junit4] 2> 2656283 INFO (jetty-closer-1526-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@36551e{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 2656283 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=25774891
[junit4] 2> 2656283 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2656283 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@b3b8d0: rootName = solr_40621, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@cc2598
[junit4] 2> 2656286 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2656286 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1426c40: rootName = solr_42065, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@74ea6c
[junit4] 2> 2656287 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:42065_solr
[junit4] 2> 2656290 INFO (jetty-closer-1526-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 2656290 INFO (zkCallback-1536-thread-4-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 2656295 INFO (coreCloseExecutor-4629-thread-1) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.c.SolrCore [forwardingCollection_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@1da43ec
[junit4] 2> 2656295 INFO (coreCloseExecutor-4629-thread-1) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.forwardingCollection.shard1.replica_n1, tag=31081452
[junit4] 2> 2656295 INFO (coreCloseExecutor-4629-thread-1) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@396ee1: rootName = solr_42065, domain = solr.core.forwardingCollection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.forwardingCollection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@40e4bd
[junit4] 2> 2656295 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2656296 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@19a5772: rootName = solr_40621, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@26b02e
[junit4] 2> 2656297 INFO (zkCallback-1537-thread-6-processing-n:127.0.0.1:40621_solr) [n:127.0.0.1:40621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 2656301 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2656302 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1e5540b: rootName = solr_40621, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@74ea6c
[junit4] 2> 2656307 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:40621_solr
[junit4] 2> 2656308 INFO (zkCallback-1537-thread-6-processing-n:127.0.0.1:40621_solr) [n:127.0.0.1:40621_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 2656308 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 2656309 INFO (coreCloseExecutor-4629-thread-1) [n:127.0.0.1:42065_solr c:forwardingCollection s:shard1 r:core_node2 x:forwardingCollection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.forwardingCollection.shard1.leader, tag=31081452
[junit4] 2> 2656310 INFO (jetty-closer-1526-thread-1) [ ] o.a.s.c.Overseer Overseer (id=99037494590046214-127.0.0.1:40621_solr-n_0000000000) closing
[junit4] 2> 2656310 INFO (OverseerStateUpdate-99037494590046214-127.0.0.1:40621_solr-n_0000000000) [n:127.0.0.1:40621_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:40621_solr
[junit4] 2> 2656310 WARN (OverseerAutoScalingTriggerThread-99037494590046214-127.0.0.1:40621_solr-n_0000000000) [n:127.0.0.1:40621_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 2656311 INFO (zkCallback-1536-thread-4-processing-n:127.0.0.1:42065_solr) [n:127.0.0.1:42065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 2656345 ERROR (Thread[Thread-2005,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:42065_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 2656418 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15fda139c7c0007, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2656419 ERROR (Thread[Thread-2007,5,TGRP-TestSolrCloudWithSecureImpersonation]) [n:127.0.0.1:40621_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 2656421 INFO (jetty-closer-1526-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@16c0e9b{/solr,null,UNAVAILABLE}
[junit4] 2> 2656421 INFO (jetty-closer-1526-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@fc9151{/solr,null,UNAVAILABLE}
[junit4] 2> 2656421 ERROR (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2656422 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43675 43675
[junit4] 2> 2661538 INFO (Thread-1988) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43675 43675
[junit4] 2> 2661538 WARN (Thread-1988) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2> 2 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/collections/forwardingCollection/state.json
[junit4] 2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
[junit4] 2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
[junit4] 2>
[junit4] 2> 2661562 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[58BB826C014095CF]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Nov 20, 2017 3:37:54 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> 2681605 ERROR (jetty-launcher-1525-thread-1-EventThread) [n:127.0.0.1:42065_solr ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
[junit4] 2> java.lang.IllegalStateException: Client is not started
[junit4] 2> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
[junit4] 2> at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:477)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceFacade.getZooKeeper(NamespaceFacade.java:112)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Nov 20, 2017 3:38:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
[junit4] 2> 1) Thread[id=15658, name=jetty-launcher-1525-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Nov 20, 2017 3:38:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=15658, name=jetty-launcher-1525-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] 2> 2681684 ERROR (jetty-launcher-1525-thread-2-EventThread) [n:127.0.0.1:40621_solr ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Nov 20, 2017 3:38:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: All leaked threads terminated.
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@9a6a01), locale=es-VE, timezone=Antarctica/South_Pole
[junit4] 2> NOTE: Linux 4.10.0-37-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=217606560,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [SmileWriterTest, TestCryptoKeys, ResponseHeaderTest, TestPullReplicaErrorHandling, AtomicUpdatesTest, NotRequiredUniqueKeyTest, TestBackupRepositoryFactory, ShowFileRequestHandlerTest, TestJavabinTupleStreamParser, ShardRoutingCustomTest, ZkSolrClientTest, TestJsonFacetRefinement, SolrGangliaReporterTest, TestConfigSetProperties, TestFaceting, UninvertDocValuesMergePolicyTest, TestLocalFSCloudBackupRestore, HdfsBasicDistributedZk2Test, SchemaVersionSpecificBehaviorTest, TestComplexPhraseLeadingWildcard, DeleteLastCustomShardedReplicaTest, TestManagedSchemaThreadSafety, TestSchemalessBufferedUpdates, BinaryUpdateRequestHandlerTest, DistributedIntervalFacetingTest, CoreAdminRequestStatusTest, TestStressLucene, BadIndexSchemaTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestDocumentBuilder, TestWriterPerf, TestOrdValues, TestFreeTextSuggestions, TestSolr4Spatial, BlockDirectoryTest, TestRestManager, OverseerModifyCollectionTest, DataDrivenBlockJoinTest, PingRequestHandlerTest, HdfsChaosMonkeySafeLeaderTest, TestReloadDeadlock, TestXIncludeConfig, TestPivotHelperCode, TestCursorMarkWithoutUniqueKey, DirectUpdateHandlerTest, DOMUtilTest, TestInPlaceUpdatesDistrib, ImplicitSnitchTest, TestCollationField, ZkFailoverTest, EnumFieldTest, HdfsUnloadDistributedZkTest, TestSha256AuthenticationProvider, SolrIndexSplitterTest, LukeRequestHandlerTest, TestFieldResource, TestLeaderElectionWithEmptyReplica, SimpleCollectionCreateDeleteTest, TestRebalanceLeaders, DebugComponentTest, TestReloadAndDeleteDocs, DocumentAnalysisRequestHandlerTest, HdfsLockFactoryTest, TestFastWriter, TestRawTransformer, TestSchemaNameResource, BadCopyFieldTest, SolrXmlInZkTest, TestHashQParserPlugin, V2StandaloneTest, TestPayloadScoreQParserPlugin, TestInfoStreamLogging, TestAtomicUpdateErrorCases, TestLazyCores, SolrRequestParserTest, TestRTGBase, RemoteQueryErrorTest, SegmentsInfoRequestHandlerTest, CdcrReplicationHandlerTest, OpenCloseCoreStressTest, CreateCollectionCleanupTest, ClassificationUpdateProcessorTest, SpellingQueryConverterTest, TestSmileRequest, TestAnalyzeInfixSuggestions, DocValuesMultiTest, PrimUtilsTest, TestOmitPositions, ChaosMonkeySafeLeaderTest, TestSolrCloudWithDelegationTokens, TestSolrConfigHandler, SolrMetricManagerTest, OverseerTaskQueueTest, MoreLikeThisHandlerTest, SortSpecParsingTest, AddBlockUpdateTest, TestStressReorder, TestExportWriter, RequestHandlersTest, TestHalfAndHalfDocValues, TestStressRecovery, CloudMLTQParserTest, BlobRepositoryCloudTest, TestAnalyzedSuggestions, MetricsHandlerTest, ExitableDirectoryReaderTest, SaslZkACLProviderTest, RegexBoostProcessorTest, TestExactSharedStatsCache, TestConfigSetsAPI, ConfigureRecoveryStrategyTest, TestScoreJoinQPNoScore, DistribDocExpirationUpdateProcessorTest, TestAuthorizationFramework, VersionInfoTest, TestCloudInspectUtil, TestSurroundQueryParser, PathHierarchyTokenizerFactoryTest, DistributedFacetPivotSmallTest, BasicDistributedZkTest, TestRandomRequestDistribution, TestReplicaProperties, TestRequestStatusCollectionAPI, TestShortCircuitedRequests, TestSolrCloudWithSecureImpersonation]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=58BB826C014095CF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es-VE -Dtests.timezone=Antarctica/South_Pole -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | TestSolrCloudWithSecureImpersonation (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
[junit4] > 1) Thread[id=15658, name=jetty-launcher-1525-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] > at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] > at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] > at __randomizedtesting.SeedInfo.seed([58BB826C014095CF]:0)
[junit4] Completed [561/750 (1!)] on J1 in 55.70s, 12 tests, 1 error <<< FAILURES!
[...truncated 43770 lines...]
[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_144) - Build #
20961 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20961/
Java: 64bit/jdk1.8.0_144 -XX:-UseCompressedOops -XX:+UseG1GC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation: 1) Thread[id=18920, name=jetty-launcher-2803-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41) at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244) at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44) at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61) at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
1) Thread[id=18920, name=jetty-launcher-2803-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
at __randomizedtesting.SeedInfo.seed([E76DAA1731158300]:0)
FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:33379/ermm/wm
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33379/ermm/wm
at __randomizedtesting.SeedInfo.seed([E76DAA1731158300:6F3995CD9FE9EEF8]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:314)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 43 more
Build Log:
[...truncated 13363 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001/init-core-data-001
[junit4] 2> 2172781 WARN (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[E76DAA1731158300]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 2172782 INFO (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[E76DAA1731158300]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 2172782 INFO (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[E76DAA1731158300]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 2172782 INFO (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[E76DAA1731158300]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 2172783 INFO (SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[E76DAA1731158300]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ermm/wm
[junit4] 2> 2172783 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.ChaosMonkeyNothingIsSafeWithPullReplicasTest Starting ChaosMonkey test with 1 shards and 4 nodes
[junit4] 2> 2172786 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2172786 INFO (Thread-4487) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2172786 INFO (Thread-4487) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2172788 ERROR (Thread-4487) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2172886 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.ZkTestServer start zk server on port:40057
[junit4] 2> 2172898 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2172899 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2172899 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2172900 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2172900 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2172900 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2172901 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2172901 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2172901 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2172902 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2172902 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2172902 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use TLOG replicas unless explicitly asked otherwise
[junit4] 2> 2172957 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 2172960 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@416c5467{/ermm/wm,null,AVAILABLE}
[junit4] 2> 2172962 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1eaa7568{HTTP/1.1,[http/1.1]}{127.0.0.1:33379}
[junit4] 2> 2172962 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.e.j.s.Server Started @2174449ms
[junit4] 2> 2172962 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001/tempDir-001/control/data, hostContext=/ermm/wm, hostPort=33379, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001/control-001/cores}
[junit4] 2> 2172962 ERROR (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2172962 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 2172962 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2172962 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2172962 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-21T04:26:55.043Z
[junit4] 2> 2172963 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2172964 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001/control-001/solr.xml
[junit4] 2> 2172967 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2172967 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2172968 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@19ce4f55, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2172972 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40057/solr
[junit4] 2> 2173002 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2173002 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33379_ermm%2Fwm
[junit4] 2> 2173003 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.Overseer Overseer (id=99040520756789252-127.0.0.1:33379_ermm%2Fwm-n_0000000000) starting
[junit4] 2> 2173007 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33379_ermm%2Fwm
[junit4] 2> 2173008 INFO (OverseerStateUpdate-99040520756789252-127.0.0.1:33379_ermm%2Fwm-n_0000000000) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2173078 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19ce4f55
[junit4] 2> 2173084 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19ce4f55
[junit4] 2> 2173084 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19ce4f55
[junit4] 2> 2173085 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001/control-001/cores
[junit4] 2> 2173099 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2173099 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40057/solr ready
[junit4] 2> 2173100 INFO (qtp894776408-19083) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33379_ermm%252Fwm&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2173102 INFO (OverseerThreadFactory-9391-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 2173205 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 2173205 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2173307 INFO (zkCallback-3004-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 2174220 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 2174228 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 2174294 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2174301 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 2174301 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@19ce4f55
[junit4] 2> 2174301 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2174301 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 2174303 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=29.3974609375, floorSegmentMB=1.7607421875, forceMergeDeletesPctAllowed=6.954751165427733, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 2174315 WARN (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2174335 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2174335 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2174337 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 2174337 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2174338 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=41, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.374093088120682]
[junit4] 2> 2174338 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@49bf7c2e[control_collection_shard1_replica_n1] main]
[junit4] 2> 2174339 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2174339 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2174339 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2174340 INFO (searcherExecutor-9394-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@49bf7c2e[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2174340 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1584648333737066496
[junit4] 2> 2174342 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2174342 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2174342 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33379/ermm/wm/control_collection_shard1_replica_n1/
[junit4] 2> 2174342 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2174342 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:33379/ermm/wm/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 2174342 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2174343 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33379/ermm/wm/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 2233157 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.ChaosMonkey monkey: stop jetty! 33379
[junit4] 2> 2233157 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=676155684
[junit4] 2> 2233157 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2233157 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e219958: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6167d2d4
[junit4] 2> 2233160 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2233160 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6783c767: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@4e1f6bec
[junit4] 2> 2233162 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2233163 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@450e775b: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@6b1e6c8a
[junit4] 2> 2233164 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:33379_ermm%2Fwm
[junit4] 2> 2233164 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 2233164 INFO (zkCallback-3004-thread-2-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 2233164 INFO (coreCloseExecutor-9399-thread-1) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@8da585d
[junit4] 2> 2233164 INFO (coreCloseExecutor-9399-thread-1) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.control_collection.shard1.replica_n1, tag=148527197
[junit4] 2> 2233164 INFO (coreCloseExecutor-9399-thread-1) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3642e62d: rootName = null, domain = solr.core.control_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.control_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@38c4e9b2
[junit4] 2> 2233169 INFO (coreCloseExecutor-9399-thread-1) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 r:core_node2 x:control_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.control_collection.shard1.leader, tag=148527197
[junit4] 2> 2244026 INFO (ScheduledTrigger-9389-thread-1) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 2244027 INFO (AutoscalingActionExecutor-9390-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.a.ExecutePlanAction No operations to execute for event: {
[junit4] 2> "id":"14f8ff631af65d80T8rq1du2o1pdy0mbigw8s059m9",
[junit4] 2> "source":".auto_add_replicas",
[junit4] 2> "eventTime":1511238476102000000,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1511238476102000000],
[junit4] 2> "_enqueue_time_":1511238486106000000,
[junit4] 2> "nodeNames":["127.0.0.1:33379_ermm%2Fwm"]}}
[junit4] 2> 2353101 ERROR (qtp894776408-19083) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: create the collection time out:180s
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:312)
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:244)
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:222)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> 2353102 INFO (qtp894776408-19083) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:33379_ermm%252Fwm&wt=javabin&version=2} status=500 QTime=180001
[junit4] 2> 2353102 ERROR (qtp894776408-19083) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: create the collection time out:180s
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:312)
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:244)
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:222)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> 2773302 ERROR (OverseerThreadFactory-9391-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:33379/ermm/wm
[junit4] 2> org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33379/ermm/wm
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
[junit4] 2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
[junit4] 2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:172)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.net.SocketTimeoutException: Read timed out
[junit4] 2> at java.net.SocketInputStream.socketRead0(Native Method)
[junit4] 2> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
[junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:171)
[junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:141)
[junit4] 2> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
[junit4] 2> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
[junit4] 2> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
[junit4] 2> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
[junit4] 2> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
[junit4] 2> at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
[junit4] 2> at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
[junit4] 2> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
[junit4] 2> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
[junit4] 2> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
[junit4] 2> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
[junit4] 2> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
[junit4] 2> ... 12 more
[junit4] 2> 2773303 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.Overseer Overseer (id=99040520756789252-127.0.0.1:33379_ermm%2Fwm-n_0000000000) closing
[junit4] 2> 2773304 ERROR (OverseerThreadFactory-9391-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [control_collection].
[junit4] 2> 2773304 INFO (OverseerStateUpdate-99040520756789252-127.0.0.1:33379_ermm%2Fwm-n_0000000000) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33379_ermm%2Fwm
[junit4] 2> 2773304 INFO (OverseerThreadFactory-9391-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 2803333 INFO (zkCallback-3004-thread-5-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [0])
[junit4] 2> 2803334 ERROR (OverseerThreadFactory-9391-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.OverseerCollectionMessageHandler Collection: control_collection operation: create failed:org.apache.solr.common.SolrException: Could not fully remove collection: control_collection
[junit4] 2> at org.apache.solr.cloud.DeleteCollectionCmd.call(DeleteCollectionCmd.java:109)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionMessageHandler.cleanupCollection(OverseerCollectionMessageHandler.java:677)
[junit4] 2> at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:302)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:231)
[junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:464)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> 2803339 INFO (OverseerThreadFactory-9391-thread-1-processing-n:127.0.0.1:33379_ermm%2Fwm) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2803340 WARN (OverseerAutoScalingTriggerThread-99040520756789252-127.0.0.1:33379_ermm%2Fwm-n_0000000000) [n:127.0.0.1:33379_ermm%2Fwm ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 2803342 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1eaa7568{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 2803342 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@416c5467{/ermm/wm,null,UNAVAILABLE}
[junit4] 2> 2803387 ERROR (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController Error getting leader from zk
[junit4] 2> org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
[junit4] 2> at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
[junit4] 2> at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
[junit4] 2> at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
[junit4] 2> at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:890)
[junit4] 2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1051)
[junit4] 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:952)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2803388 ERROR (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkContainer :org.apache.solr.common.SolrException: Error getting leader from zk for shard shard1
[junit4] 2> at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1207)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
[junit4] 2> at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
[junit4] 2> at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
[junit4] 2> at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:890)
[junit4] 2> at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1051)
[junit4] 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:952)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: org.apache.solr.common.SolrException: No registered leader was found after waiting for 1220000ms , collection: control_collection slice: shard1 saw state=null with live_nodes=[]
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
[junit4] 2> at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
[junit4] 2> ... 41 more
[junit4] 2>
[junit4] 2> 2803395 INFO (qtp894776408-19085) [n:127.0.0.1:33379_ermm%2Fwm c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=630190
[junit4] 2> 2803395 ERROR (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2803396 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[E76DAA1731158300]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40057 40057
[junit4] 2> 2803429 INFO (Thread-4487) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40057 40057
[junit4] 2> 2803430 WARN (Thread-4487) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2> 2 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeWithPullReplicasTest -Dtests.method=test -Dtests.seed=E76DAA1731158300 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pl -Dtests.timezone=Europe/Bratislava -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 631s J1 | ChaosMonkeyNothingIsSafeWithPullReplicasTest.test <<<
[junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:33379/ermm/wm
[junit4] > at __randomizedtesting.SeedInfo.seed([E76DAA1731158300:6F3995CD9FE9EEF8]:0)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
[junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
[junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:314)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > Caused by: java.net.SocketTimeoutException: Read timed out
[junit4] > at java.net.SocketInputStream.socketRead0(Native Method)
[junit4] > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:171)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:141)
[junit4] > at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
[junit4] > at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
[junit4] > at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
[junit4] > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
[junit4] > at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
[junit4] > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
[junit4] > at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
[junit4] > at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
[junit4] > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
[junit4] > at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
[junit4] > at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
[junit4] > ... 43 more
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_E76DAA1731158300-001
[junit4] 2> Nov 21, 2017 4:37:25 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=76, maxMBSortInHeap=6.964072937679399, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3c0ffe42), locale=pl, timezone=Europe/Bratislava
[junit4] 2> NOTE: Linux 4.10.0-37-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=179381456,total=532676608
[junit4] 2> NOTE: All tests run in this JVM: [SolrIndexConfigTest, BadIndexSchemaTest, BinaryUpdateRequestHandlerTest, TestFastWriter, TestComplexPhraseLeadingWildcard, NodeLostTriggerTest, TestDistributedStatsComponentCardinality, TestLegacyField, TestNumericRangeQuery64, ShardRoutingCustomTest, PathHierarchyTokenizerFactoryTest, ZkFailoverTest, GraphQueryTest, TestQuerySenderNoQuery, HdfsUnloadDistributedZkTest, ClusterStateUpdateTest, LargeFieldTest, TestSolrConfigHandlerConcurrent, RestartWhileUpdatingTest, TestSolr4Spatial, TestFieldResource, OverseerModifyCollectionTest, MoveReplicaTest, TestRTGBase, SolrIndexSplitterTest, TestRebalanceLeaders, SignatureUpdateProcessorFactoryTest, DataDrivenBlockJoinTest, HLLUtilTest, TestDynamicFieldCollectionResource, LukeRequestHandlerTest, AtomicUpdatesTest, TestChildDocTransformer, TestNumericTerms32, SolrCmdDistributorTest, TestPayloadScoreQParserPlugin, V2StandaloneTest, SaslZkACLProviderTest, TestRequestStatusCollectionAPI, TestCryptoKeys, LeaderInitiatedRecoveryOnShardRestartTest, CoreAdminRequestStatusTest, TestShortCircuitedRequests, TestNoOpRegenerator, MetricsHandlerTest, TestPivotHelperCode, TestStressRecovery, ConvertedLegacyTest, TestCollectionAPIs, TestDocSet, ShowFileRequestHandlerTest, TestWriterPerf, DistribDocExpirationUpdateProcessorTest, TermVectorComponentDistributedTest, SpatialHeatmapFacetsTest, MBeansHandlerTest, TestObjectReleaseTracker, HighlighterConfigTest, DistributedDebugComponentTest, DeleteLastCustomShardedReplicaTest, TestClusterProperties, TestFastLRUCache, HdfsChaosMonkeySafeLeaderTest, TestMaxScoreQueryParser, NumericFieldsTest, ResponseHeaderTest, TestReloadAndDeleteDocs, AddReplicaTest, TestBackupRepositoryFactory, TestJmxIntegration, PolyFieldTest, TestSmileRequest, BlobRepositoryCloudTest, TestLegacyTerms, TestAuthorizationFramework, ExplicitHLLTest, TestReplicationHandlerBackup, ChaosMonkeySafeLeaderWithPullReplicasTest, TestFieldCacheReopen, DistributedVersionInfoTest, TestCollationField, CloudExitableDirectoryReaderTest, HdfsAutoAddReplicasIntegrationTest, TestOmitPositions, TestCloudInspectUtil, TestPullReplicaErrorHandling, ExitableDirectoryReaderTest, TestAnalyzeInfixSuggestions, DebugComponentTest, TestEmbeddedSolrServerAdminHandler, SmileWriterTest, InfixSuggestersTest, SuggesterTSTTest, TestCustomStream, ChaosMonkeySafeLeaderTest, CurrencyFieldTypeTest, TestReloadDeadlock, PingRequestHandlerTest, MissingSegmentRecoveryTest, TestDownShardTolerantSearch, CreateCollectionCleanupTest, TestMacroExpander, DistanceUnitsTest, TestClusterStateMutator, SegmentsInfoRequestHandlerTest, SOLR749Test, TestStressLucene, ClassificationUpdateProcessorTest, BadCopyFieldTest, NotRequiredUniqueKeyTest, TestFaceting, ZkCLITest, TestAnalyzedSuggestions, HighlighterMaxOffsetTest, TestConfigSetsAPI, RequestHandlersTest, TestJettySolrRunner, TestRangeQuery, CloudMLTQParserTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestTlogReplica, TestJsonFacetRefinement, RegexBoostProcessorTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, TestRandomFaceting, ZkSolrClientTest, TestZkChroot, TestDistributedSearch, TestDistributedGrouping, TestHashPartitioner, ZkControllerTest, TestRealTimeGet, TestStressReorder, TestReload, TestStressVersions, DistributedTermsComponentTest, SimpleFacetsTest, SolrCoreTest, StatsComponentTest, TestGroupingSearch, QueryElevationComponentTest, PeerSyncTest, TestSort, TestFiltering, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, SimplePostToolTest, TestCoreDiscovery, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPolicy1, CacheHeaderTest, TestSurroundQueryParser, TestQueryUtils, DocumentAnalysisRequestHandlerTest, TestQueryTypes, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, TermVectorComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, FastVectorHighlighterTest, SolrPluginUtilsTest, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, CopyFieldTest, ResponseLogComponentTest, TestSolrDeletionPolicy2, SampleTest, TestBinaryField, TestSearchPerf, MinimalSchemaTest, TestElisionMultitermQuery, OutputWriterTest, TestSolrCoreProperties, TestLuceneMatchVersion, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, OpenExchangeRatesOrgProviderTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, SpellingQueryConverterTest, DOMUtilTest, RAMDirectoryFactoryTest, ClusterStateTest, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, CircularListTest, DistributedMLTComponentTest, CursorPagingTest, DistributedIntervalFacetingTest, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestLuceneIndexBackCompat, TestEmbeddedSolrServerConstructors, ConnectionReuseTest, AssignTest, AsyncCallRequestStatusResponseTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] Completed [520/750 (1!)] on J1 in 631.17s, 1 test, 1 error <<< FAILURES!
[...truncated 128 lines...]
[junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_E76DAA1731158300-001/init-core-data-001
[junit4] 2> 3164497 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[E76DAA1731158300]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 3164497 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[E76DAA1731158300]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 3164497 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[E76DAA1731158300]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 3164507 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[E76DAA1731158300]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_E76DAA1731158300-001/tempDir-001
[junit4] 2> 3164507 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[E76DAA1731158300]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 3164507 INFO (Thread-5804) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 3164507 INFO (Thread-5804) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 3164508 ERROR (Thread-5804) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3164607 INFO (SUITE-TestSolrCloudWithSecureImpersonation-seed#[E76DAA1731158300]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:42463
[junit4] 2> 3164615 INFO (jetty-launcher-2803-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3164615 INFO (jetty-launcher-2803-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3164617 INFO (jetty-launcher-2803-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16313fcb{/solr,null,AVAILABLE}
[junit4] 2> 3164617 INFO (jetty-launcher-2803-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@69efc832{/solr,null,AVAILABLE}
[junit4] 2> 3164618 INFO (jetty-launcher-2803-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@13775053{HTTP/1.1,[http/1.1]}{127.0.0.1:33943}
[junit4] 2> 3164618 INFO (jetty-launcher-2803-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@742232f6{HTTP/1.1,[http/1.1]}{127.0.0.1:45905}
[junit4] 2> 3164618 INFO (jetty-launcher-2803-thread-1) [ ] o.e.j.s.Server Started @3166170ms
[junit4] 2> 3164618 INFO (jetty-launcher-2803-thread-2) [ ] o.e.j.s.Server Started @3166171ms
[junit4] 2> 3164618 INFO (jetty-launcher-2803-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33943}
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45905}
[junit4] 2> 3164619 ERROR (jetty-launcher-2803-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3164619 ERROR (jetty-launcher-2803-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-21T04:43:26.761Z
[junit4] 2> 3164619 INFO (jetty-launcher-2803-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-21T04:43:26.761Z
[junit4] 2> 3164621 INFO (jetty-launcher-2803-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3164621 INFO (jetty-launcher-2803-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3164630 INFO (jetty-launcher-2803-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42463/solr
[junit4] 2> 3164631 INFO (jetty-launcher-2803-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42463/solr
[junit4] 2> 3164680 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3164681 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45905_solr
[junit4] 2> 3164681 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.Overseer Overseer (id=99040585754214405-127.0.0.1:45905_solr-n_0000000000) starting
[junit4] 2> 3164682 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3164683 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33943_solr
[junit4] 2> 3164684 INFO (zkCallback-2814-thread-1-processing-n:127.0.0.1:45905_solr) [n:127.0.0.1:45905_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3164684 INFO (zkCallback-2815-thread-1-processing-n:127.0.0.1:33943_solr) [n:127.0.0.1:33943_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3164692 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45905_solr
[junit4] 2> 3164695 INFO (zkCallback-2814-thread-1-processing-n:127.0.0.1:45905_solr) [n:127.0.0.1:45905_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3164695 INFO (zkCallback-2815-thread-1-processing-n:127.0.0.1:33943_solr) [n:127.0.0.1:33943_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3164711 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 3164711 WARN (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3164711 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 3164712 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 3164712 INFO (jetty-launcher-2803-thread-2-EventThread) [n:127.0.0.1:45905_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 3164755 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 3164755 WARN (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3164755 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.s.KerberosPlugin Params: {solr.kerberos.impersonator.user.noGroups.hosts=*, solr.kerberos.impersonator.user.bogusGroup.groups=__some_bogus_group, token.valid=30, cookie.domain=127.0.0.1, solr.kerberos.impersonator.user.anyHostAnyUser.groups=*, solr.kerberos.impersonator.user.bogusGroup.hosts=*, solr.kerberos.impersonator.user.anyHostAnyUser.hosts=*, token.validity=36000, solr.kerberos.impersonator.user.noHosts.groups=*, type=kerberos, solr.kerberos.impersonator.user.localHostAnyGroup.groups=*, delegation-token.token-kind=solr-dt, solr.kerberos.impersonator.user.wrongHost.hosts=1.1.1.1.1.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, solr.kerberos.impersonator.user.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, signer.secret.provider.zookeeper.path=/token, solr.kerberos.impersonator.user.anyHostUsersGroup.groups=jenkins, zk-dt-secret-manager.enable=true, solr.kerberos.impersonator.user.wrongHost.groups=*, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper, solr.kerberos.impersonator.user.anyHostUsersGroup.hosts=*}
[junit4] 2> 3164756 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 3164757 INFO (jetty-launcher-2803-thread-1-EventThread) [n:127.0.0.1:33943_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 3164764 WARN (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3164788 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33943.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56dc639e
[junit4] 2> 3164793 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33943.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56dc639e
[junit4] 2> 3164793 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33943.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56dc639e
[junit4] 2> 3164794 INFO (jetty-launcher-2803-thread-1) [n:127.0.0.1:33943_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_E76DAA1731158300-001/tempDir-001/node1/.
[junit4] 2> 3165725 WARN (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config'
[junit4] 2> 3165757 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45905.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56dc639e
[junit4] 2> 3165763 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45905.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56dc639e
[junit4] 2> 3165763 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45905.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@56dc639e
[junit4] 2> 3165764 INFO (jetty-launcher-2803-thread-2) [n:127.0.0.1:45905_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestSolrCloudWithSecureImpersonation_E76DAA1731158300-001/tempDir-001/node2/.
[junit4] 2> 3165794 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[E76DAA1731158300]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyUnknownRemote
[junit4] 2> 3169596 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyUnknownRemote-seed#[E76DAA1731158300]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyUnknownRemote
[junit4] 2> 3169616 INFO (TEST-TestSolrCloudWithSecureImpersonation.testProxyNoConfigGroups-seed#[E76DAA1731158300]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
[junit4] 2> 3
[...truncated too long message...]
105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Nov 21, 2017 4:44:16 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=18920, name=jetty-launcher-2803-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] 2> 3214782 ERROR (jetty-launcher-2803-thread-1-EventThread) [n:127.0.0.1:33943_solr ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Nov 21, 2017 4:44:16 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: All leaked threads terminated.
[junit4] 2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2e742d4b), locale=mt, timezone=Etc/Universal
[junit4] 2> NOTE: Linux 4.10.0-37-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=187193648,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [AutoAddReplicasIntegrationTest, HdfsRecoveryZkTest, RegexBytesRefFilterTest, TestFuzzyAnalyzedSuggestions, DocValuesNotIndexedTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestStandardQParsers, TestTrackingShardHandlerFactory, TestCharFilters, ChaosMonkeyNothingIsSafeTest, TestCloudJSONFacetJoinDomain, TestSearcherReuse, SearchRateTriggerTest, TestSubQueryTransformer, BigEndianAscendingWordSerializerTest, MultiThreadedOCPTest, AlternateDirectoryTest, TestFieldCacheWithThreads, DistributedExpandComponentTest, TestLockTree, TestConfigReload, SolrTestCaseJ4Test, LeaderElectionTest, TestDeleteCollectionOnDownNodes, HdfsThreadLeakTest, TestExpandComponent, DistanceFunctionTest, DistributedFacetPivotLongTailTest, TestReqParamsAPI, TestSizeLimitedDistributedMap, PropertiesRequestHandlerTest, StatsReloadRaceTest, CdcrBootstrapTest, ConjunctionSolrSpellCheckerTest, FieldMutatingUpdateProcessorTest, DistributedSuggestComponentTest, MoveReplicaHDFSFailoverTest, TestMissingGroups, TestLeaderElectionZkExpiry, SuggesterTest, ManagedSchemaRoundRobinCloudTest, TestTrie, TestDistributedMissingSort, DirectSolrSpellCheckerTest, MergeStrategyTest, DeleteNodeTest, TestLRUCache, SolrSlf4jReporterTest, AnalyticsQueryTest, HttpPartitionTest, HdfsWriteToMultipleCollectionsTest, TestManagedSynonymFilterFactory, TestNonDefinedSimilarityFactory, TestSSLRandomization, TestNumericTerms64, ComputePlanActionTest, TestNumericTokenStream, TimeZoneUtilsTest, TestFieldCacheSort, TestCloudNestedDocsSort, TestLFUCache, ScriptEngineTest, TestSortByMinMaxFunction, AliasIntegrationTest, DeleteStatusTest, PeerSyncWithIndexFingerprintCachingTest, TestPushWriter, TriggerIntegrationTest, IndexSchemaTest, TestSolrCloudWithKerberosAlt, TestCoreContainer, TestTolerantSearch, RequiredFieldsTest, TriLevelCompositeIdRoutingTest, LoggingHandlerTest, LeaderFailureAfterFreshStartTest, PeerSyncReplicationTest, DefaultValueUpdateProcessorTest, TestShardHandlerFactory, TestSQLHandlerNonCloud, AutoscalingHistoryHandlerTest, BlockJoinFacetRandomTest, TestSolrXml, NumberUtilsTest, TestHighFrequencyDictionaryFactory, TestBinaryResponseWriter, XsltUpdateRequestHandlerTest, WrapperMergePolicyFactoryTest, DisMaxRequestHandlerTest, TestFilteredDocIdSet, HttpTriggerListenerTest, TestStressInPlaceUpdates, TestXmlQParser, RecoveryZkTest, TestSolr4Spatial2, CdcrRequestHandlerTest, IndexBasedSpellCheckerTest, TestExactStatsCache, SimpleMLTQParserTest, TestValueSourceCache, SuggestComponentContextFilterQueryTest, SystemInfoHandlerTest, SpellPossibilityIteratorTest, DistributedSpellCheckComponentTest, TestBulkSchemaAPI, BasicAuthStandaloneTest, TestOverriddenPrefixQueryForCustomFieldType, TestCloudRecovery, CoreMergeIndexesAdminHandlerTest, TestLMDirichletSimilarityFactory, TestReplicaProperties, BadComponentTest, ParsingFieldUpdateProcessorsTest, HdfsCollectionsAPIDistributedZkTest, SolrGraphiteReporterTest, TestQuerySenderListener, WordBreakSolrSpellCheckerTest, DirectSolrConnectionTest, TestSystemCollAutoCreate, PluginInfoTest, TestDefaultStatsCache, UtilsToolTest, ZkStateReaderTest, UnloadDistributedZkTest, LegacyCloudClusterPropTest, MoveReplicaHDFSTest, OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, RecoveryAfterSoftCommitTest, SharedFSAutoReplicaFailoverTest, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestCollectionAPI, TestCollectionsAPIViaSolrCloudCluster, TestExclusionRuleCollectionAccess, TestHdfsCloudBackupRestore, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestRequestForwarding, TestSegmentSorting, TestSolrCloudWithDelegationTokens, TestSolrCloudWithSecureImpersonation]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithSecureImpersonation -Dtests.seed=E76DAA1731158300 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt -Dtests.timezone=Etc/Universal -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J2 | TestSolrCloudWithSecureImpersonation (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation:
[junit4] > 1) Thread[id=18920, name=jetty-launcher-2803-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestSolrCloudWithSecureImpersonation]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] > at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] > at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] > at __randomizedtesting.SeedInfo.seed([E76DAA1731158300]:0)
[junit4] Completed [560/750 (2!)] on J2 in 50.30s, 12 tests, 1 error <<< FAILURES!
[...truncated 47915 lines...]
[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_144) - Build #
20960 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20960/
Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC
5 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.DistribJoinFromCollectionTest
Error Message:
Error from server at https://127.0.0.1:37573/solr: create the collection time out:180s
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:37573/solr: create the collection time out:180s
at __randomizedtesting.SeedInfo.seed([5DC95CE425E6FA15]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.DistribJoinFromCollectionTest.setupCluster(DistribJoinFromCollectionTest.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ForceLeaderTest
Error Message:
24 threads leaked from SUITE scope at org.apache.solr.cloud.ForceLeaderTest: 1) Thread[id=14339, name=zkCallback-2131-thread-4, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=14301, name=qtp27995-14301, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=14327, name=searcherExecutor-5945-thread-1, state=WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=14313, name=TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[5DC95CE425E6FA15]-SendThread(127.0.0.1:46183), state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 5) Thread[id=14333, name=qtp27995-14333, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=14302, name=qtp27995-14302, state=RUNNABLE, group=TGRP-ForceLeaderTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=14335, name=qtp27995-14335, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=14338, name=zkCallback-2131-thread-3, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=14334, name=qtp27995-14334, state=RUNNABLE, group=TGRP-ForceLeaderTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=14314, name=TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[5DC95CE425E6FA15]-EventThread, state=WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 11) Thread[id=14300, name=qtp27995-14300-acceptor-0@1fefc6f-ServerConnector@13916a0{HTTP/1.1,[http/1.1]}{127.0.0.1:38211}, state=RUNNABLE, group=TGRP-ForceLeaderTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=14299, name=qtp27995-14299, state=RUNNABLE, group=TGRP-ForceLeaderTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=14298, name=qtp27995-14298, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 14) Thread[id=14316, name=Thread-3218, state=WAITING, group=TGRP-ForceLeaderTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1723) 15) Thread[id=14321, name=Scheduler-15365980, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 16) Thread[id=14304, name=org.eclipse.jetty.server.session.HashSessionManager@190020eTimer, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 17) Thread[id=14315, name=zkCallback-2131-thread-1-processing-n:127.0.0.1:32925_kxxm, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:357) at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:771) at org.apache.solr.cloud.Overseer.getCompletedMap(Overseer.java:676) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.<init>(Overseer.java:111) at org.apache.solr.cloud.Overseer.start(Overseer.java:518) at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:799) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135) at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56) at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348) at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$0(SolrZkClient.java:262) at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$224/1365071.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$101/23679871.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18) Thread[id=14303, name=qtp27995-14303, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:767) at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178) at org.apache.solr.cloud.ZkController.register(ZkController.java:1079) at org.apache.solr.cloud.ZkController.register(ZkController.java:1014) at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181) at org.apache.solr.core.ZkContainer$$Lambda$178/32367201.run(Unknown Source) at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208) at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:890) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1051) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:952) at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91) at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$66/17506016.execute(Unknown Source) at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384) at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177) at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735) at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 19) Thread[id=14309, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 20) Thread[id=14337, name=zkCallback-2131-thread-2, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21) Thread[id=14341, name=zkCallback-2131-thread-6, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 22) Thread[id=14308, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 23) Thread[id=14296, name=qtp27995-14296, state=RUNNABLE, group=TGRP-ForceLeaderTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 24) Thread[id=14340, name=zkCallback-2131-thread-5, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 24 threads leaked from SUITE scope at org.apache.solr.cloud.ForceLeaderTest:
1) Thread[id=14339, name=zkCallback-2131-thread-4, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=14301, name=qtp27995-14301, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=14327, name=searcherExecutor-5945-thread-1, state=WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=14313, name=TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[5DC95CE425E6FA15]-SendThread(127.0.0.1:46183), state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
5) Thread[id=14333, name=qtp27995-14333, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=14302, name=qtp27995-14302, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=14335, name=qtp27995-14335, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=14338, name=zkCallback-2131-thread-3, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
9) Thread[id=14334, name=qtp27995-14334, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=14314, name=TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[5DC95CE425E6FA15]-EventThread, state=WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
11) Thread[id=14300, name=qtp27995-14300-acceptor-0@1fefc6f-ServerConnector@13916a0{HTTP/1.1,[http/1.1]}{127.0.0.1:38211}, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
12) Thread[id=14299, name=qtp27995-14299, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
13) Thread[id=14298, name=qtp27995-14298, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
14) Thread[id=14316, name=Thread-3218, state=WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1723)
15) Thread[id=14321, name=Scheduler-15365980, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
16) Thread[id=14304, name=org.eclipse.jetty.server.session.HashSessionManager@190020eTimer, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
17) Thread[id=14315, name=zkCallback-2131-thread-1-processing-n:127.0.0.1:32925_kxxm, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:357)
at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:771)
at org.apache.solr.cloud.Overseer.getCompletedMap(Overseer.java:676)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.<init>(Overseer.java:111)
at org.apache.solr.cloud.Overseer.start(Overseer.java:518)
at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:799)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$0(SolrZkClient.java:262)
at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$224/1365071.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$101/23679871.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
18) Thread[id=14303, name=qtp27995-14303, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:767)
at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
at org.apache.solr.cloud.ZkController.register(ZkController.java:1079)
at org.apache.solr.cloud.ZkController.register(ZkController.java:1014)
at org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
at org.apache.solr.core.ZkContainer$$Lambda$178/32367201.run(Unknown Source)
at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:208)
at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:890)
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1051)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:952)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:91)
at org.apache.solr.handler.admin.CoreAdminOperation$$Lambda$66/17506016.execute(Unknown Source)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:497)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
19) Thread[id=14309, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
20) Thread[id=14337, name=zkCallback-2131-thread-2, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
21) Thread[id=14341, name=zkCallback-2131-thread-6, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22) Thread[id=14308, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
23) Thread[id=14296, name=qtp27995-14296, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
24) Thread[id=14340, name=zkCallback-2131-thread-5, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([5DC95CE425E6FA15]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ForceLeaderTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=14316, name=Thread-3218, state=WAITING, group=TGRP-ForceLeaderTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1723) 2) Thread[id=14339, name=zkCallback-2131-thread-4, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=14321, name=Scheduler-15365980, state=WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=14304, name=org.eclipse.jetty.server.session.HashSessionManager@190020eTimer, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=14315, name=zkCallback-2131-thread-1, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=14327, name=searcherExecutor-5945-thread-1, state=WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=14303, name=qtp27995-14303, state=RUNNABLE, group=TGRP-ForceLeaderTest] at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=14313, name=TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[5DC95CE425E6FA15]-SendThread(127.0.0.1:46183), state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 9) Thread[id=14302, name=qtp27995-14302, state=RUNNABLE, group=TGRP-ForceLeaderTest] at java.lang.Thread.isInterrupted(Native Method) at java.lang.Thread.isInterrupted(Thread.java:964) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:218) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=14338, name=zkCallback-2131-thread-3, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=14337, name=zkCallback-2131-thread-2, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=14341, name=zkCallback-2131-thread-6, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=14334, name=qtp27995-14334, state=RUNNABLE, group=TGRP-ForceLeaderTest] at java.lang.Thread.isInterrupted(Native Method) at java.lang.Thread.isInterrupted(Thread.java:964) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:218) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 14) Thread[id=14299, name=qtp27995-14299, state=RUNNABLE, group=TGRP-ForceLeaderTest] at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 15) Thread[id=14340, name=zkCallback-2131-thread-5, state=TIMED_WAITING, group=TGRP-ForceLeaderTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=14316, name=Thread-3218, state=WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1723)
2) Thread[id=14339, name=zkCallback-2131-thread-4, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=14321, name=Scheduler-15365980, state=WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=14304, name=org.eclipse.jetty.server.session.HashSessionManager@190020eTimer, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=14315, name=zkCallback-2131-thread-1, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=14327, name=searcherExecutor-5945-thread-1, state=WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=14303, name=qtp27995-14303, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:207)
at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=14313, name=TEST-ForceLeaderTest.testReplicasInLIRNoLeader-seed#[5DC95CE425E6FA15]-SendThread(127.0.0.1:46183), state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
9) Thread[id=14302, name=qtp27995-14302, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at java.lang.Thread.isInterrupted(Native Method)
at java.lang.Thread.isInterrupted(Thread.java:964)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:218)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=14338, name=zkCallback-2131-thread-3, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
11) Thread[id=14337, name=zkCallback-2131-thread-2, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
12) Thread[id=14341, name=zkCallback-2131-thread-6, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
13) Thread[id=14334, name=qtp27995-14334, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at java.lang.Thread.isInterrupted(Native Method)
at java.lang.Thread.isInterrupted(Thread.java:964)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:218)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:92)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
14) Thread[id=14299, name=qtp27995-14299, state=RUNNABLE, group=TGRP-ForceLeaderTest]
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:104)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
15) Thread[id=14340, name=zkCallback-2131-thread-5, state=TIMED_WAITING, group=TGRP-ForceLeaderTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([5DC95CE425E6FA15]:0)
FAILED: org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:32925/kxxm
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:32925/kxxm
at __randomizedtesting.SeedInfo.seed([5DC95CE425E6FA15:BB5E68241C640374]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:417)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:333)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 44 more
FAILED: org.apache.solr.cloud.TestCloudPivotFacet.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:43187/jz/vz
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:43187/jz/vz
at __randomizedtesting.SeedInfo.seed([5DC95CE425E6FA15:D59D633E8B1A97ED]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:314)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:
[...truncated too long message...]
with live_nodes=[]
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:771)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:736)
[junit4] 2> at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1178)
[junit4] 2> ... 41 more
[junit4] 2>
[junit4] 2> 2781493 INFO (qtp4698545-12652) [n:127.0.0.1:43187_jz%2Fvz c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=630209
[junit4] 2> 2781493 ERROR (TEST-TestCloudPivotFacet.test-seed#[5DC95CE425E6FA15]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2781494 INFO (TEST-TestCloudPivotFacet.test-seed#[5DC95CE425E6FA15]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34569 34569
[junit4] 2> 2787111 INFO (Thread-1930) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34569 34569
[junit4] 2> 2787111 WARN (Thread-1930) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/clusterstate.json
[junit4] 2> 2 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 2 /solr/live_nodes
[junit4] 2> 2 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestCloudPivotFacet -Dtests.method=test -Dtests.seed=5DC95CE425E6FA15 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hi-IN -Dtests.timezone=America/Sao_Paulo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 637s J0 | TestCloudPivotFacet.test <<<
[junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:43187/jz/vz
[junit4] > at __randomizedtesting.SeedInfo.seed([5DC95CE425E6FA15:D59D633E8B1A97ED]:0)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
[junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
[junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:314)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > Caused by: java.net.SocketTimeoutException: Read timed out
[junit4] > at java.net.SocketInputStream.socketRead0(Native Method)
[junit4] > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:171)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:141)
[junit4] > at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
[junit4] > at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
[junit4] > at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
[junit4] > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
[junit4] > at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
[junit4] > at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
[junit4] > at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
[junit4] > at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
[junit4] > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
[junit4] > at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
[junit4] > at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
[junit4] > ... 43 more
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudPivotFacet_5DC95CE425E6FA15-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1345, maxMBSortInHeap=5.481762355374785, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@11d320b), locale=hi-IN, timezone=America/Sao_Paulo
[junit4] 2> NOTE: Linux 4.10.0-37-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=140778632,total=330895360
[junit4] 2> NOTE: All tests run in this JVM: [TestFieldCollectionResource, TestBulkSchemaConcurrent, AtomicUpdateProcessorFactoryTest, HttpSolrCallGetCoreTest, HdfsRestartWhileUpdatingTest, CollectionStateFormat2Test, TestJsonRequest, TestFoldingMultitermQuery, HLLSerializationTest, SoftAutoCommitTest, SolrCoreMetricManagerTest, PKIAuthenticationIntegrationTest, TestNestedDocsSort, SyncSliceTest, TestNRTOpen, TestRawResponseWriter, V2ApiIntegrationTest, TestRecovery, UpdateLogTest, TestLegacyNumericRangeQueryBuilder, TestSlowCompositeReaderWrapper, TestReversedWildcardFilterFactory, TestCollapseQParserPlugin, SpellCheckComponentTest, TestLegacyNumericUtils, TestNumericRangeQuery32, BasicDistributedZkTest, DateFieldTest, TestSolrCoreParser, TestHighlightDedupGrouping, TestRandomCollapseQParserPlugin, TestLRUStatsCache, TestRandomRequestDistribution, DistribCursorPagingTest, CachingDirectoryFactoryTest, TestBadConfig, ExecutePlanActionTest, ActionThrottleTest, TestSolrCloudWithSecureImpersonation, TestCopyFieldCollectionResource, SubstringBytesRefFilterTest, BlockJoinFacetSimpleTest, TestDFRSimilarityFactory, PreAnalyzedUpdateProcessorTest, ClassificationUpdateProcessorIntegrationTest, TestRestoreCore, TestSolrDeletionPolicy1, TermsComponentTest, DocValuesTest, TestSort, CollectionsAPISolrJTest, CdcrVersionReplicationTest, IndexSchemaRuntimeFieldTest, SharedFSAutoReplicaFailoverTest, TestFieldCache, SuggestComponentTest, UpdateParamsTest, TestSolrQueryResponse, TestFileDictionaryLookup, SolrCloudReportersTest, AnalysisAfterCoreReloadTest, SpatialFilterTest, JsonLoaderTest, TestRequestForwarding, DeleteInactiveReplicaTest, TestXmlQParserPlugin, TestCorePropertiesReload, TestSweetSpotSimilarityFactory, MetricUtilsTest, FileUtilsTest, NoCacheHeaderTest, DocExpirationUpdateProcessorFactoryTest, SpellCheckCollatorWithCollapseTest, TestJoin, NodeAddedTriggerTest, TestManagedResourceStorage, BasicZkTest, SliceStateTest, TestCollationFieldDocValues, ResponseLogComponentTest, TestApiFramework, PeerSyncTest, TestReplicationHandler, TestGraphTermsQParserPlugin, TestLuceneIndexBackCompat, TestStressUserVersions, TestTolerantUpdateProcessorRandomCloud, InfoHandlerTest, HdfsChaosMonkeyNothingIsSafeTest, TestCustomSort, DistributedQueueTest, TestFastOutputStream, DirectUpdateHandlerOptimizeTest, TermVectorComponentTest, TestPhraseSuggestions, TestComponentsName, TestWordDelimiterFilterFactory, TestEmbeddedSolrServerSchemaAPI, ZkControllerTest, UpdateRequestProcessorFactoryTest, TestConfig, CdcrUpdateLogTest, TestSolrIndexConfig, FastVectorHighlighterTest, MinimalSchemaTest, TestMultiWordSynonyms, TestCoreAdminApis, DistributedFacetExistsSmallTest, TestCrossCoreJoin, SolrCoreTest, UUIDUpdateProcessorFallbackTest, TestNamedUpdateProcessors, TestPerFieldSimilarityWithDefaultOverride, CacheHeaderTest, TestCloudPivotFacet]
[junit4] Completed [615/750 (3!)] on J0 in 636.56s, 1 test, 1 error <<< FAILURES!
[...truncated 43510 lines...]
[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_144) - Build #
20959 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20959/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseParallelGC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestV2Request
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request: 1) Thread[id=2607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:
1) Thread[id=2607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([D3F71A5565FD9D61]:0)
FAILED: org.apache.solr.client.solrj.request.TestV2Request.testCloudSolrClient
Error Message:
Error from server at https://127.0.0.1:43371/solr: Could not fully remove collection: test
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: Error from server at https://127.0.0.1:43371/solr: Could not fully remove collection: test
at __randomizedtesting.SeedInfo.seed([D3F71A5565FD9D61:4A0191E5C04E150A]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:829)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:620)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.client.solrj.request.TestV2Request.assertSuccess(TestV2Request.java:42)
at org.apache.solr.client.solrj.request.TestV2Request.doTest(TestV2Request.java:91)
at org.apache.solr.client.solrj.request.TestV2Request.testCloudSolrClient(TestV2Request.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.client.solrj.request.TestV2Request.testHttpSolrClient
Error Message:
Error from server at https://127.0.0.1:43371/solr: Could not fully create collection: test
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: Error from server at https://127.0.0.1:43371/solr: Could not fully create collection: test
at __randomizedtesting.SeedInfo.seed([D3F71A5565FD9D61:BEF027FC4C98F46]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:829)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:620)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.client.solrj.request.TestV2Request.assertSuccess(TestV2Request.java:42)
at org.apache.solr.client.solrj.request.TestV2Request.doTest(TestV2Request.java:72)
at org.apache.solr.client.solrj.request.TestV2Request.testHttpSolrClient(TestV2Request.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.index.hdfs.CheckHdfsIndexTest.doTest
Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:38103
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:38103
at __randomizedtesting.SeedInfo.seed([5F0B528D3EE88E6E:F84FEA2953539DD7]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:314)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:991)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 43 more
Build Log:
[...truncated 12686 lines...]
[junit4] Suite: org.apache.solr.index.hdfs.CheckHdfsIndexTest
[junit4] 2> 749248 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/init-core-data-001
[junit4] 2> 749248 WARN (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=27 numCloses=27
[junit4] 2> 749249 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 749250 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 749251 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 749364 WARN (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 749376 WARN (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 749379 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 749396 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_localdomain_45665_hdfs____.6bdck2/webapp
[junit4] 2> 749840 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:45665
[junit4] 2> 749922 WARN (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 749923 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 749940 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_42483_datanode____ejb8t4/webapp
[junit4] 2> 750363 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42483
[junit4] 2> 750414 WARN (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 750415 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 750423 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46331_datanode____7c9rp2/webapp
[junit4] 2> 750476 ERROR (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/tempDir-001/hdfsBaseDir/data/data2/]] heartbeating to localhost.localdomain/127.0.0.1:46153) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 750481 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x6e1ad142be193: from storage DS-1f6637b3-b83a-4c73-b5b6-353872931025 node DatanodeRegistration(127.0.0.1:42449, datanodeUuid=e64c4ef8-7e87-4464-ba62-537f3c05d2cc, infoPort=35755, infoSecurePort=0, ipcPort=45675, storageInfo=lv=-56;cid=testClusterID;nsid=891163725;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 750481 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x6e1ad142be193: from storage DS-a38281d4-f6cd-4313-a918-539ddc87f9c1 node DatanodeRegistration(127.0.0.1:42449, datanodeUuid=e64c4ef8-7e87-4464-ba62-537f3c05d2cc, infoPort=35755, infoSecurePort=0, ipcPort=45675, storageInfo=lv=-56;cid=testClusterID;nsid=891163725;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 750854 INFO (SUITE-CheckHdfsIndexTest-seed#[5F0B528D3EE88E6E]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46331
[junit4] 2> 750975 ERROR (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/tempDir-001/hdfsBaseDir/data/data4/]] heartbeating to localhost.localdomain/127.0.0.1:46153) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 750985 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x6e1ad32218e0d: from storage DS-889a6dce-737b-46c2-9cd6-b7924f60ac37 node DatanodeRegistration(127.0.0.1:45225, datanodeUuid=c0724f08-85fe-41d5-8d90-43462acc9e0b, infoPort=42679, infoSecurePort=0, ipcPort=43247, storageInfo=lv=-56;cid=testClusterID;nsid=891163725;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
[junit4] 2> 750986 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x6e1ad32218e0d: from storage DS-1021dd01-e164-4da6-a184-1646d994a7e0 node DatanodeRegistration(127.0.0.1:45225, datanodeUuid=c0724f08-85fe-41d5-8d90-43462acc9e0b, infoPort=42679, infoSecurePort=0, ipcPort=43247, storageInfo=lv=-56;cid=testClusterID;nsid=891163725;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 751130 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 751131 INFO (Thread-2235) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 751131 INFO (Thread-2235) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 751132 ERROR (Thread-2235) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 751231 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.ZkTestServer start zk server on port:37175
[junit4] 2> 751243 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 751245 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 751246 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 751247 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 751248 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 751249 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 751250 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 751251 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 751252 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 751253 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 751254 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 751257 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 751328 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 751336 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@191b9ebd{/,null,AVAILABLE}
[junit4] 2> 751337 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4c7fc1e4{SSL,[ssl, http/1.1]}{127.0.0.1:35275}
[junit4] 2> 751337 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.Server Started @752702ms
[junit4] 2> 751337 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost.localdomain:46153/hdfs__localhost.localdomain_46153__home_jenkins_workspace_Lucene-Solr-master-Linux_solr_build_solr-core_test_J0_temp_solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001_tempDir-002_control_data, replicaType=NRT, hostContext=/, hostPort=35275, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/control-001/cores}
[junit4] 2> 751337 ERROR (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 751337 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 751337 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 751337 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 751337 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T21:00:36.030Z
[junit4] 2> 751348 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 751348 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/control-001/solr.xml
[junit4] 2> 751351 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 751352 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 751352 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 751356 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37175/solr
[junit4] 2> 751385 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 751385 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35275_
[junit4] 2> 751386 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.c.Overseer Overseer (id=99038765765361669-127.0.0.1:35275_-n_0000000000) starting
[junit4] 2> 751389 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35275_
[junit4] 2> 751389 INFO (zkCallback-1026-thread-1-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 751422 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 751427 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 751427 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 751428 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:35275_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/control-001/cores
[junit4] 2> 751438 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 751438 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37175/solr ready
[junit4] 2> 751449 INFO (qtp74200346-7870) [n:127.0.0.1:35275_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35275_&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 751450 INFO (OverseerThreadFactory-3350-thread-1-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 751556 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 751556 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 751658 INFO (zkCallback-1026-thread-1-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 752564 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 752851 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 752914 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 752920 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 752921 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 752921 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost.localdomain:46153/solr_hdfs_home
[junit4] 2> 752921 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 752921 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 752921 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[hdfs://localhost.localdomain:46153/solr_hdfs_home/control_collection/core_node2/data/]
[junit4] 2> 752922 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost.localdomain:46153/solr_hdfs_home/control_collection/core_node2/data/snapshot_metadata
[junit4] 2> 752926 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 752926 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 752926 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
[junit4] 2> 752932 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 752933 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost.localdomain:46153/solr_hdfs_home/control_collection/core_node2/data
[junit4] 2> 752946 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost.localdomain:46153/solr_hdfs_home/control_collection/core_node2/data/index
[junit4] 2> 752950 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 752950 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 752953 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 752953 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=50, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6962516258501444]
[junit4] 2> 752966 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 752967 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741825_1001 size 69
[junit4] 2> 752972 WARN (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 752999 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 752999 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 752999 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 753005 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 753005 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 753007 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=69.36328125, floorSegmentMB=1.6416015625, forceMergeDeletesPctAllowed=26.4579807769865, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8137663584887416
[junit4] 2> 753013 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@72f931d1[control_collection_shard1_replica_n1] main]
[junit4] 2> 753013 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 753014 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 753014 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 753015 INFO (searcherExecutor-3353-thread-1-processing-n:127.0.0.1:35275_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@72f931d1[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 753015 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1584620254221303808
[junit4] 2> 753017 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 753017 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 753017 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:35275/control_collection_shard1_replica_n1/
[junit4] 2> 753017 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 753017 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:35275/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 753017 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 753018 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:35275/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 753119 INFO (zkCallback-1026-thread-1-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 753169 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 753178 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1622
[junit4] 2> 753182 INFO (qtp74200346-7870) [n:127.0.0.1:35275_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 753272 INFO (zkCallback-1026-thread-1-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 753453 INFO (OverseerCollectionConfigSetProcessor-99038765765361669-127.0.0.1:35275_-n_0000000000) [n:127.0.0.1:35275_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 754182 INFO (qtp74200346-7870) [n:127.0.0.1:35275_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35275_&wt=javabin&version=2} status=0 QTime=2733
[junit4] 2> 754188 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 754189 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37175/solr ready
[junit4] 2> 754189 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 754194 INFO (qtp74200346-7869) [n:127.0.0.1:35275_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 754200 INFO (OverseerThreadFactory-3350-thread-2-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 754201 WARN (OverseerThreadFactory-3350-thread-2-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 754405 INFO (qtp74200346-7869) [n:127.0.0.1:35275_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 754405 INFO (qtp74200346-7869) [n:127.0.0.1:35275_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=211
[junit4] 2> 754475 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/shard-1-001 of type NRT
[junit4] 2> 754476 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@dea75a4{/,null,AVAILABLE}
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@8048c6a{SSL,[ssl, http/1.1]}{127.0.0.1:38553}
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.e.j.s.Server Started @755843ms
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost.localdomain:46153/hdfs__localhost.localdomain_46153__home_jenkins_workspace_Lucene-Solr-master-Linux_solr_build_solr-core_test_J0_temp_solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001_tempDir-002_jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=38553, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/shard-1-001/cores}
[junit4] 2> 754478 ERROR (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 754478 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T21:00:39.171Z
[junit4] 2> 754480 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 754480 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/shard-1-001/solr.xml
[junit4] 2> 754482 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 754482 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 754482 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 754484 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37175/solr
[junit4] 2> 754488 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:38553_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 754489 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:38553_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 754489 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:38553_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38553_
[junit4] 2> 754490 INFO (zkCallback-1039-thread-1-processing-n:127.0.0.1:38553_) [n:127.0.0.1:38553_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 754490 INFO (zkCallback-1034-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 754490 INFO (zkCallback-1026-thread-1-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 754534 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:38553_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 754539 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:38553_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 754539 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:38553_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 754540 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [n:127.0.0.1:38553_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/shard-1-001/cores
[junit4] 2> 754554 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:38553_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 754555 INFO (OverseerCollectionConfigSetProcessor-99038765765361669-127.0.0.1:35275_-n_0000000000) [n:127.0.0.1:35275_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 754555 INFO (OverseerThreadFactory-3350-thread-3-processing-n:127.0.0.1:35275_) [n:127.0.0.1:35275_ ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:38553_ for creating new replica
[junit4] 2> 754569 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 754569 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 755583 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 755593 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema [collection1_shard1_replica_n21] Schema name=test
[junit4] 2> 755659 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 755665 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n21' using configuration from collection collection1, trusted=true
[junit4] 2> 755666 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n21' (registry 'solr.core.collection1.shard1.replica_n21') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@729f1d10
[junit4] 2> 755666 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost.localdomain:46153/solr_hdfs_home
[junit4] 2> 755666 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 755666 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 755666 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [[collection1_shard1_replica_n21] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.index.hdfs.CheckHdfsIndexTest_5F0B528D3EE88E6E-001/shard-1-001/cores/collection1_shard1_replica_n21], dataDir=[hdfs://localhost.localdomain:46153/solr_hdfs_home/collection1/core_node22/data/]
[junit4] 2> 755667 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost.localdomain:46153/solr_hdfs_home/collection1/core_node22/data/snapshot_metadata
[junit4] 2> 755671 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 755671 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 755673 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 755674 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost.localdomain:46153/solr_hdfs_home/collection1/core_node22/data
[junit4] 2> 755691 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost.localdomain:46153/solr_hdfs_home/collection1/core_node22/data/index
[junit4] 2> 755695 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
[junit4] 2> 755695 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
[junit4] 2> 755697 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 755697 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=50, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6962516258501444]
[junit4] 2> 755711 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 755712 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 755716 WARN (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 755742 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 755742 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 755742 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
[junit4] 2> 755748 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 755748 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 755749 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=69.36328125, floorSegmentMB=1.6416015625, forceMergeDeletesPctAllowed=26.4579807769865, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8137663584887416
[junit4] 2> 755753 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.SolrIndexSearcher Opening [Searcher@95f6aae[collection1_shard1_replica_n21] main]
[junit4] 2> 755754 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 755754 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 755755 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 755755 INFO (searcherExecutor-3364-thread-1-processing-n:127.0.0.1:38553_ x:collection1_shard1_replica_n21 s:shard1 c:collection1) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SolrCore [collection1_shard1_replica_n21] Registered new searcher Searcher@95f6aae[collection1_shard1_replica_n21] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 755755 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1584620257094402048
[junit4] 2> 755758 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 755758 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 755758 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38553/collection1_shard1_replica_n21/
[junit4] 2> 755758 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 755758 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.SyncStrategy https://127.0.0.1:38553/collection1_shard1_replica_n21/ has no replicas
[junit4] 2> 755758 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 755759 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38553/collection1_shard1_replica_n21/ shard1
[junit4] 2> 755909 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 755918 INFO (qtp1261935280-7927) [n:127.0.0.1:38553_ c:collection1 s:shard1 x:collection1_shard1_replica_n21] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n21&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1349
[junit4] 2> 755920 INFO (qtp74200346-7864) [n:127.0.0.1:35275_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:38553_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1366
[junit4] 2> 755923 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testChecksumsOnlyVerbose
[junit4] 2> 756052 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW]]} size 0
[junit4] 2> 756052 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756061 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 756061 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 756068 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756069 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741830_1006 size 8093
[junit4] 2> 756073 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW]]} size 0
[junit4] 2> 756073 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741831_1007 size 313
[junit4] 2> 756079 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW]]} size 0
[junit4] 2> 756079 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741832_1008 size 70
[junit4] 2> 756083 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW]]} size 0
[junit4] 2> 756084 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741833_1009 size 402
[junit4] 2> 756088 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW]]} size 0
[junit4] 2> 756088 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW]]} size 0
[junit4] 2> 756092 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741835_1011{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW]]} size 97
[junit4] 2> 756092 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741835_1011 size 97
[junit4] 2> 756497 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW]]} size 0
[junit4] 2> 756498 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741827_1003 size 80554
[junit4] 2> 756505 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 756507 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741836_1012 size 100
[junit4] 2> 756535 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756536 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756545 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW]]} size 0
[junit4] 2> 756545 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW]]} size 0
[junit4] 2> 756551 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW]]} size 0
[junit4] 2> 756551 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756555 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756556 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756556 INFO (OverseerCollectionConfigSetProcessor-99038765765361669-127.0.0.1:35275_-n_0000000000) [n:127.0.0.1:35275_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 756560 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756560 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|FINALIZED]]} size 0
[junit4] 2> 756605 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 756605 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 756607 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW]]} size 0
[junit4] 2> 756609 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW], ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW]]} size 0
[junit4] 2> 756614 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a38281d4-f6cd-4313-a918-539ddc87f9c1:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-1021dd01-e164-4da6-a184-1646d994a7e0:NORMAL:127.0.0.1:45225|RBW]]} size 0
[junit4] 2> 756614 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741844_1020 size 384
[junit4] 2> 756615 INFO (IPC Server handler 0 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741828_1004 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 756616 INFO (IPC Server handler 2 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741835_1011 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756616 INFO (IPC Server handler 4 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741834_1010 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756617 INFO (IPC Server handler 3 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741836_1012 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756617 INFO (IPC Server handler 6 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741832_1008 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 756617 INFO (IPC Server handler 5 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741830_1006 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 756618 INFO (IPC Server handler 7 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741827_1003 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 756618 INFO (IPC Server handler 9 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741840_1016 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756619 INFO (IPC Server handler 8 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741839_1015 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 756619 INFO (IPC Server handler 1 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741837_1013 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756620 INFO (IPC Server handler 0 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741833_1009 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756620 INFO (IPC Server handler 2 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741831_1007 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756620 INFO (IPC Server handler 4 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741838_1014 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 756621 INFO (IPC Server handler 3 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741829_1005 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 756621 INFO (IPC Server handler 6 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741841_1017 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 756629 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45225 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 756629 INFO (Block report processor) [ ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42449 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-1f6637b3-b83a-4c73-b5b6-353872931025:NORMAL:127.0.0.1:42449|RBW], ReplicaUC[[DISK]DS-889a6dce-737b-46c2-9cd6-b7924f60ac37:NORMAL:127.0.0.1:45225|FINALIZED]]} size 0
[junit4] 2> 757302 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost.localdomain:46153/solr
[junit4] 2> 757303 INFO (IPC Server handler 8 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741843_1019 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 757303 INFO (IPC Server handler 8 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741842_1018 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 757303 INFO (IPC Server handler 8 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741844_1020 127.0.0.1:45225 127.0.0.1:42449
[junit4] 2> 757303 INFO (IPC Server handler 8 on 46153) [ ] BlockStateChange BLOCK* addToInvalidates: blk_1073741845_1021 127.0.0.1:42449 127.0.0.1:45225
[junit4] 2> 757304 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testChecksumsOnlyVerbose
[junit4] 2> 757304 INFO (TEST-CheckHdfsIndexTest.testChecksumsOnlyVerbose-seed#[5F0B528D3EE88E6E]) [ ] o.a.s
[...truncated too long message...]
07-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2e0b2ba3{/solr,null,UNAVAILABLE}
[junit4] 2> 211650 INFO (zkCallback-519-thread-3-processing-n:127.0.0.1:42599_solr) [n:127.0.0.1:42599_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 211650 INFO (jetty-closer-507-thread-2) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:35785_solr
[junit4] 2> 211650 INFO (jetty-closer-507-thread-4) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:42599_solr
[junit4] 2> 211651 INFO (jetty-closer-507-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 211651 INFO (jetty-closer-507-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 211651 INFO (zkCallback-527-thread-3-processing-n:127.0.0.1:41149_solr) [n:127.0.0.1:41149_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
[junit4] 2> 211651 INFO (jetty-closer-507-thread-4) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 211651 INFO (jetty-closer-507-thread-3) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1d809b0a: rootName = solr_41149, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@44495357
[junit4] 2> 211651 INFO (zkCallback-519-thread-2-processing-n:127.0.0.1:42599_solr) [n:127.0.0.1:42599_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (1)
[junit4] 2> 211651 INFO (zkCallback-527-thread-1-processing-n:127.0.0.1:41149_solr) [n:127.0.0.1:41149_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 211651 INFO (zkCallback-528-thread-4-processing-n:127.0.0.1:35785_solr) [n:127.0.0.1:35785_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (1)
[junit4] 2> 211653 INFO (jetty-closer-507-thread-3) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:41149_solr
[junit4] 2> 211653 INFO (zkCallback-528-thread-4-processing-n:127.0.0.1:35785_solr) [n:127.0.0.1:35785_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 211653 INFO (zkCallback-527-thread-1-processing-n:127.0.0.1:41149_solr) [n:127.0.0.1:41149_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 211653 INFO (zkCallback-519-thread-3-processing-n:127.0.0.1:42599_solr) [n:127.0.0.1:42599_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 211653 INFO (jetty-closer-507-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 211657 INFO (jetty-closer-507-thread-2) [ ] o.a.s.c.Overseer Overseer (id=99038907842166796-127.0.0.1:35785_solr-n_0000000000) closing
[junit4] 2> 211657 INFO (OverseerStateUpdate-99038907842166796-127.0.0.1:35785_solr-n_0000000000) [n:127.0.0.1:35785_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35785_solr
[junit4] 2> 211658 INFO (jetty-closer-507-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5b591d9e{/solr,null,UNAVAILABLE}
[junit4] 2> 211658 INFO (jetty-closer-507-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@fa3f7d4{/solr,null,UNAVAILABLE}
[junit4] 2> 211660 WARN (OverseerAutoScalingTriggerThread-99038907842166796-127.0.0.1:35785_solr-n_0000000000) [n:127.0.0.1:35785_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 211665 INFO (jetty-closer-507-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@50cd98f6{/solr,null,UNAVAILABLE}
[junit4] 2> 211665 ERROR (SUITE-TestV2Request-seed#[D3F71A5565FD9D61]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 211665 INFO (SUITE-TestV2Request-seed#[D3F71A5565FD9D61]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41257 41257
[junit4] 2> 218206 INFO (Thread-622) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41257 41257
[junit4] 2> 218206 WARN (Thread-622) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 8 /solr/configs/config
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 28 /solr/collections/test/state.json
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2> 2 /solr/overseer_elect/election/99038907842166796-127.0.0.1:35785_solr-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.request.TestV2Request_D3F71A5565FD9D61-001
[junit4] 2> Nov 20, 2017 9:38:23 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> Nov 20, 2017 9:38:43 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:
[junit4] 2> 1) Thread[id=2607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Nov 20, 2017 9:38:43 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=2607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] 2> Nov 20, 2017 9:38:43 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: All leaked threads terminated.
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1688, maxMBSortInHeap=6.905576848619933, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@3e8c494a), locale=pl-PL, timezone=Australia/Lindeman
[junit4] 2> NOTE: Linux 4.10.0-37-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=249117880,total=527958016
[junit4] 2> NOTE: All tests run in this JVM: [TestCloudSolrClientConnections, LessThanEqualToEvaluatorTest, CloudSolrClientBuilderTest, JettyWebappTest, DocumentAnalysisResponseTest, RegressionEvaluatorTest, NormalizeEvaluatorTest, UuidEvaluatorTest, HttpSolrClientBuilderTest, TestJavaBinCodec, ExclusiveOrEvaluatorTest, GetByIdTest, TestCollectionStateWatchers, AddEvaluatorTest, SelectWithEvaluatorsTest, JdbcDriverTest, TestCollectionAdminRequest, TestJsonRecordReader, SineEvaluatorTest, DivideEvaluatorTest, StreamExpressionToExplanationTest, SolrExceptionTest, ArrayEvaluatorTest, ModifiableSolrParamsTest, SolrExampleEmbeddedTest, LargeVolumeJettyTest, SolrParamTest, TestEmbeddedSolrServer, TestNamedListCodec, TestDocumentObjectBinder, TestHash, ContentStreamTest, SolrDocumentTest, ClientUtilsTest, NamedListTest, AnlysisResponseBaseTest, TestSolrJErrorHandling, ConcurrentUpdateSolrClientBuilderTest, LBHttpSolrClientBuilderTest, JDBCStreamTest, StreamExpressionToExpessionTest, StreamingTest, AbsoluteValueEvaluatorTest, AndEvaluatorTest, ArcCosineEvaluatorTest, ArcSineEvaluatorTest, ArcTangentEvaluatorTest, CeilingEvaluatorTest, CoalesceEvaluatorTest, CorrelationEvaluatorTest, CosineEvaluatorTest, CubedRootEvaluatorTest, CumulativeProbabilityEvaluatorTest, EmpiricalDistributionEvaluatorTest, EqualToEvaluatorTest, FieldValueEvaluatorTest, GreaterThanEqualToEvaluatorTest, GreaterThanEvaluatorTest, HyperbolicCosineEvaluatorTest, HyperbolicSineEvaluatorTest, LessThanEvaluatorTest, MultiplyEvaluatorTest, PowerEvaluatorTest, SubtractEvaluatorTest, TangentEvaluatorTest, UniformDistributionEvaluatorTest, StreamExpressionParserTest, OperationsTest, SchemaTest, TestV2Request]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestV2Request -Dtests.seed=D3F71A5565FD9D61 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pl-PL -Dtests.timezone=Australia/Lindeman -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J2 | TestV2Request (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestV2Request:
[junit4] > 1) Thread[id=2607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([D3F71A5565FD9D61]:0)
[junit4] Completed [146/146 (1!)] on J2 in 120.20s, 3 tests, 3 errors <<< FAILURES!
[...truncated 46352 lines...]
[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-10-ea+32) - Build
# 20958 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20958/
Java: 64bit/jdk-10-ea+32 -XX:-UseCompressedOops -XX:+UseParallelGC
12 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MoveReplicaHDFSFailoverTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.cloud.MoveReplicaHDFSFailoverTest.setupClass(MoveReplicaHDFSFailoverTest.java:55)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MoveReplicaHDFSTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.cloud.MoveReplicaHDFSTest.setupClass(MoveReplicaHDFSTest.java:44)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.setupClass(HdfsAutoAddReplicasIntegrationTest.java:42)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:67)
at org.apache.solr.cloud.hdfs.HdfsRecoverLeaseTest.beforeClass(HdfsRecoverLeaseTest.java:50)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.setupClass(HdfsRecoveryZkTest.java:41)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsThreadLeakTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsThreadLeakTest.beforeClass(HdfsThreadLeakTest.java:45)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.core.HdfsDirectoryFactoryTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:67)
at org.apache.solr.core.HdfsDirectoryFactoryTest.setupClass(HdfsDirectoryFactoryTest.java:59)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestHdfsBackupRestoreCore
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.handler.TestHdfsBackupRestoreCore.setupClass(TestHdfsBackupRestoreCore.java:108)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.index.hdfs.CheckHdfsIndexTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.index.hdfs.CheckHdfsIndexTest.setupClass(CheckHdfsIndexTest.java:62)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.search.TestRecoveryHdfs
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.search.TestRecoveryHdfs.beforeClass(TestRecoveryHdfs.java:81)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: junit.framework.TestSuite.org.apache.solr.store.hdfs.HdfsDirectoryTest
Error Message:
Timed out waiting for Mini HDFS Cluster to start
Stack Trace:
java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
at org.apache.solr.store.hdfs.HdfsDirectoryTest.beforeClass(HdfsDirectoryTest.java:60)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: org.apache.lucene.search.similarities.TestBasicModelIn.testRandomScoring
Error Message:
score(1.2418185E9,19)=1.86272768E9 < score(1.2418185E9,20)=1.86272781E9
Stack Trace:
java.lang.AssertionError: score(1.2418185E9,19)=1.86272768E9 < score(1.2418185E9,20)=1.86272781E9
at __randomizedtesting.SeedInfo.seed([6E906950CE6423DE:E50F30E2D413C5D4]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.search.similarities.BaseSimilarityTestCase.doTestScoring(BaseSimilarityTestCase.java:423)
at org.apache.lucene.search.similarities.BaseSimilarityTestCase.testRandomScoring(BaseSimilarityTestCase.java:355)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 673 lines...]
[junit4] Suite: org.apache.lucene.search.similarities.TestBasicModelIn
[junit4] 1> 1.86272768E9 = score(DFRSimilarity, doc=0, freq=1.2418185E9), computed from:
[junit4] 1> 1.1342137E26 = NormalizationH1, computed from:
[junit4] 1> 1.2418185E9 = tf
[junit4] 1> 1.2267671E9 = avgFieldLength
[junit4] 1> 19.0 = len
[junit4] 1> 3.4026414E26 = BasicModelIn, computed from:
[junit4] 1> 11.0 = numberOfDocuments
[junit4] 1> 1.0 = docFreq
[junit4] 1> 5.4743582E-18 = AfterEffectB, computed from:
[junit4] 1> 1.1342137E26 = tfn
[junit4] 1> 1.2418185E9 = totalTermFreq
[junit4] 1> 1.0 = docFreq
[junit4] 1>
[junit4] 1> 1.86272781E9 = score(DFRSimilarity, doc=0, freq=1.2418185E9), computed from:
[junit4] 1> 1.077503E26 = NormalizationH1, computed from:
[junit4] 1> 1.2418185E9 = tf
[junit4] 1> 1.2267671E9 = avgFieldLength
[junit4] 1> 20.0 = len
[junit4] 1> 3.232509E26 = BasicModelIn, computed from:
[junit4] 1> 11.0 = numberOfDocuments
[junit4] 1> 1.0 = docFreq
[junit4] 1> 5.7624826E-18 = AfterEffectB, computed from:
[junit4] 1> 1.077503E26 = tfn
[junit4] 1> 1.2418185E9 = totalTermFreq
[junit4] 1> 1.0 = docFreq
[junit4] 1>
[junit4] 1> DFR I(n)B1
[junit4] 1> field="field",maxDoc=11,docCount=11,sumTotalTermFreq=13494438321,sumDocFreq=11
[junit4] 1> term="term",docFreq=1,totalTermFreq=1241818494
[junit4] 1> norm=20 (doc length ~ 20)
[junit4] 1> freq=1.2418185E9
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBasicModelIn -Dtests.method=testRandomScoring -Dtests.seed=6E906950CE6423DE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=os-RU -Dtests.timezone=SystemV/CST6 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 7.65s J2 | TestBasicModelIn.testRandomScoring <<<
[junit4] > Throwable #1: java.lang.AssertionError: score(1.2418185E9,19)=1.86272768E9 < score(1.2418185E9,20)=1.86272781E9
[junit4] > at __randomizedtesting.SeedInfo.seed([6E906950CE6423DE:E50F30E2D413C5D4]:0)
[junit4] > at org.apache.lucene.search.similarities.BaseSimilarityTestCase.doTestScoring(BaseSimilarityTestCase.java:423)
[junit4] > at org.apache.lucene.search.similarities.BaseSimilarityTestCase.testRandomScoring(BaseSimilarityTestCase.java:355)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {field=PostingsFormat(name=Direct)}, docValues:{}, maxPointsInLeafNode=57, maxMBSortInHeap=7.8104328346848835, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@54d6d8bc), locale=os-RU, timezone=SystemV/CST6
[junit4] 2> NOTE: Linux 4.10.0-37-generic amd64/Oracle Corporation 10-ea (64-bit)/cpus=8,threads=1,free=155183440,total=536346624
[junit4] 2> NOTE: All tests run in this JVM: [Test2BNumericDocValues, TestDocValuesScoring, TestDocValuesRewriteMethod, TestTermVectors, TestRamUsageEstimator, TestPerFieldPostingsFormat2, TestEarlyTerminatingSortingCollector, TestSpans, TestAllFilesHaveCodecHeader, TestBasics, TestHighCompressionMode, TestConjunctions, MultiCollectorTest, TestReadOnlyIndex, TestSortedSetDocValues, TestIndexWriterMaxDocs, TestPrefixQuery, TestIndexSearcher, TestCachingTokenFilter, TestIndexWriterDeleteByQuery, TestBooleanSimilarity, TestBufferedIndexInput, TestBytesRefArray, TestBagOfPostings, TestStopFilter, TestIntroSorter, TestCollectionUtil, TestLongRangeFieldQueries, TestInPlaceMergeSorter, TestGeoEncodingUtils, TestStressAdvance, TestIndexWriterCommit, TestDirectoryReader, TestSegmentReader, TestDisjunctionMaxQuery, TestNRTReaderWithThreads, TestSynonymQuery, TestRegExp, TestAttributeSource, TestRollingBuffer, TestExceedMaxTermLength, TestIndexCommit, TestSubScorerFreqs, TestDateSort, TestIndexWriterReader, TestTermRangeQuery, TestVersion, TestIndependenceStandardized, TestFixedBitSet, TestLucene60PointsFormat, TestAllFilesHaveChecksumFooter, TestComplexExplanationsOfNonMatches, TestFieldCacheRewriteMethod, TestNoMergePolicy, TestSearchWithThreads, TestLogMergePolicy, TestSetOnce, TestSearchForDuplicates, TestAxiomaticSimilarity, TestFastCompressionMode, TestSimpleExplanationsOfNonMatches, TestBlockPostingsFormat, TestSimilarityBase, TestParallelCompositeReader, TestBinaryDocument, TestBufferedChecksum, TestSameTokenSamePosition, TestLazyProxSkipping, TestMergePolicyWrapper, TestTopDocsCollector, TestMultiLevelSkipList, TestBasicModelIn]
[junit4] Completed [140/476 (1!)] on J2 in 7.70s, 1 test, 1 failure <<< FAILURES!
[...truncated 11752 lines...]
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSFailoverTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/init-core-data-001
[junit4] 2> 823072 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 823073 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
[junit4] 2> 823074 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 823074 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-001
[junit4] 2> 823074 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 823074 INFO (Thread-1444) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 823074 INFO (Thread-1444) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 823083 ERROR (Thread-1444) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 823174 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:41169
[junit4] 2> 823194 INFO (jetty-launcher-1303-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 823205 INFO (jetty-launcher-1303-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 823218 INFO (jetty-launcher-1303-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@771d50d1{/solr,null,AVAILABLE}
[junit4] 2> 823222 INFO (jetty-launcher-1303-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@a010be4{SSL,[ssl, http/1.1]}{127.0.0.1:34777}
[junit4] 2> 823223 INFO (jetty-launcher-1303-thread-2) [ ] o.e.j.s.Server Started @824619ms
[junit4] 2> 823223 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34777}
[junit4] 2> 823223 ERROR (jetty-launcher-1303-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 823223 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 823223 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 823223 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 823223 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T18:03:55.370464Z
[junit4] 2> 823227 INFO (jetty-launcher-1303-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@334f6afa{/solr,null,AVAILABLE}
[junit4] 2> 823230 INFO (jetty-launcher-1303-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4aff0e28{SSL,[ssl, http/1.1]}{127.0.0.1:41667}
[junit4] 2> 823230 INFO (jetty-launcher-1303-thread-1) [ ] o.e.j.s.Server Started @824627ms
[junit4] 2> 823231 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41667}
[junit4] 2> 823231 ERROR (jetty-launcher-1303-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 823231 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 823231 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 823231 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 823231 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T18:03:55.378456Z
[junit4] 2> 823234 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 823234 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 823244 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41169/solr
[junit4] 2> 823252 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41169/solr
[junit4] 2> 823343 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 823348 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41667_solr
[junit4] 2> 823349 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.c.Overseer Overseer (id=99038071044308996-127.0.0.1:41667_solr-n_0000000000) starting
[junit4] 2> 823352 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 823353 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34777_solr
[junit4] 2> 823382 INFO (zkCallback-1313-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 823384 INFO (zkCallback-1316-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 823391 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41667_solr
[junit4] 2> 823392 INFO (zkCallback-1316-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 823421 INFO (zkCallback-1313-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 823457 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34777.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@360ceaa
[junit4] 2> 823467 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41667.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@360ceaa
[junit4] 2> 823470 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34777.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@360ceaa
[junit4] 2> 823470 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34777.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@360ceaa
[junit4] 2> 823471 INFO (jetty-launcher-1303-thread-2) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-001/node2/.
[junit4] 2> 823474 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41667.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@360ceaa
[junit4] 2> 823474 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41667.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@360ceaa
[junit4] 2> 823475 INFO (jetty-launcher-1303-thread-1) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-001/node1/.
[junit4] 2> 823515 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 823515 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41169/solr ready
[junit4] 2> 823846 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 824348 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 824591 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
[junit4] 2> 824612 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 824823 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log jetty-6.1.x
[junit4] 2> 824931 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_localdomain_40695_hdfs____.qqmp8w/webapp
[junit4] 2> 825208 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:40695
[junit4] 2> 826329 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 826337 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log jetty-6.1.x
[junit4] 2> 826360 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37129_datanode____.wrw8g5/webapp
[junit4] 2> 826507 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37129
[junit4] 2> 826536 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] i.n.u.i.PlatformDependent Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
[junit4] 2> 827198 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 827200 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log jetty-6.1.x
[junit4] 2> 827225 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Extract jar:file:/home/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37493_datanode____clwpch/webapp
[junit4] 2> 827358 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37493
[junit4] 2> 827445 WARN (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data2/]] heartbeating to localhost.localdomain/127.0.0.1:34769) [ ] o.a.h.h.s.d.DataNode Unexpected exception in block pool Block pool <registering> (Datanode Uuid unassigned) service to localhost.localdomain/127.0.0.1:34769
[junit4] 2> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: .BP-1178396193-2a01:4f8:10b:2ab:0:0:0:2-1511201036305.crc
[junit4] 2> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
[junit4] 2> at org.apache.hadoop.fs.Path.<init>(Path.java:171)
[junit4] 2> at org.apache.hadoop.fs.Path.<init>(Path.java:93)
[junit4] 2> at org.apache.hadoop.fs.ChecksumFileSystem.getChecksumFile(ChecksumFileSystem.java:90)
[junit4] 2> at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:488)
[junit4] 2> at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:508)
[junit4] 2> at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:140)
[junit4] 2> at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:156)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.makeBlockPoolDataDir(DataStorage.java:598)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:465)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:573)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1386)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1351)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:216)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:637)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> Caused by: java.net.URISyntaxException: Relative path in absolute URI: .BP-1178396193-2a01:4f8:10b:2ab:0:0:0:2-1511201036305.crc
[junit4] 2> at java.base/java.net.URI.checkPath(URI.java:1941)
[junit4] 2> at java.base/java.net.URI.<init>(URI.java:757)
[junit4] 2> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
[junit4] 2> ... 17 more
[junit4] 2> 827470 WARN (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data2/]] heartbeating to localhost.localdomain/127.0.0.1:34769) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to localhost.localdomain/127.0.0.1:34769
[junit4] 2> 827627 WARN (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data4/]] heartbeating to localhost.localdomain/127.0.0.1:34769) [ ] o.a.h.h.s.d.DataNode Unexpected exception in block pool Block pool <registering> (Datanode Uuid unassigned) service to localhost.localdomain/127.0.0.1:34769
[junit4] 2> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: .BP-1178396193-2a01:4f8:10b:2ab:0:0:0:2-1511201036305.crc
[junit4] 2> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
[junit4] 2> at org.apache.hadoop.fs.Path.<init>(Path.java:171)
[junit4] 2> at org.apache.hadoop.fs.Path.<init>(Path.java:93)
[junit4] 2> at org.apache.hadoop.fs.ChecksumFileSystem.getChecksumFile(ChecksumFileSystem.java:90)
[junit4] 2> at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:488)
[junit4] 2> at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:508)
[junit4] 2> at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:140)
[junit4] 2> at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:156)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.makeBlockPoolDataDir(DataStorage.java:598)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:465)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:573)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1386)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1351)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:216)
[junit4] 2> at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:637)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> Caused by: java.net.URISyntaxException: Relative path in absolute URI: .BP-1178396193-2a01:4f8:10b:2ab:0:0:0:2-1511201036305.crc
[junit4] 2> at java.base/java.net.URI.checkPath(URI.java:1941)
[junit4] 2> at java.base/java.net.URI.<init>(URI.java:757)
[junit4] 2> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
[junit4] 2> ... 17 more
[junit4] 2> 827627 WARN (DataNode: [[[DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.MoveReplicaHDFSFailoverTest_F927525CAC20F3CB-001/tempDir-002/hdfsBaseDir/data/data4/]] heartbeating to localhost.localdomain/127.0.0.1:34769) [ ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to localhost.localdomain/127.0.0.1:34769
[junit4] 2> 827962 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster BPOfferService in datanode DataNode{data=null, localName='127.0.0.1:35271', datanodeUuid='null', xmitsInProgress=0} failed to connect to namenode at localhost.localdomain/127.0.0.1:34769
[junit4] 2> 828067 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 829067 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 830068 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 831068 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 832068 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 833068 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 834068 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 835068 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 836069 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 837069 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 838069 WARN (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Waiting for the Mini HDFS Cluster to start...
[junit4] 2> 839069 ERROR (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.h.h.MiniDFSCluster Timed out waiting for Mini HDFS Cluster to start
[junit4] 2> 839082 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 839194 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
[junit4] 2> 839307 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@a010be4{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@4aff0e28{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1403369928
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1430358015
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4d91fb2f: rootName = solr_34777, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@5a1a6b2b
[junit4] 2> 839434 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@27f5f170: rootName = solr_41667, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6bf36417
[junit4] 2> 839441 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 839441 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3d19dc31: rootName = solr_34777, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@301e247a
[junit4] 2> 839441 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 839441 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@a4564b7: rootName = solr_41667, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@301e247a
[junit4] 2> 839445 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 839445 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@12058ad1: rootName = solr_34777, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@504491d7
[junit4] 2> 839446 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 839446 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4d60a11d: rootName = solr_41667, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@504491d7
[junit4] 2> 839446 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:34777_solr
[junit4] 2> 839447 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 839447 INFO (jetty-closer-1304-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 839447 INFO (zkCallback-1316-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 839448 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:41667_solr
[junit4] 2> 839448 INFO (jetty-closer-1304-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@771d50d1{/solr,null,UNAVAILABLE}
[junit4] 2> 839448 INFO (zkCallback-1313-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 839449 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 839449 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 839449 INFO (zkCallback-1313-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 839449 INFO (jetty-closer-1304-thread-2) [ ] o.a.s.c.Overseer Overseer (id=99038071044308996-127.0.0.1:41667_solr-n_0000000000) closing
[junit4] 2> 839449 INFO (OverseerStateUpdate-99038071044308996-127.0.0.1:41667_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41667_solr
[junit4] 2> 839449 WARN (OverseerAutoScalingTriggerThread-99038071044308996-127.0.0.1:41667_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 839451 INFO (jetty-closer-1304-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@334f6afa{/solr,null,UNAVAILABLE}
[junit4] 2> 839452 ERROR (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 839452 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41169 41169
[junit4] 2> 844567 INFO (Thread-1444) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41169 41169
[junit4] 2> 844567 WARN (Thread-1444) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2> 2 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> 844567 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41169 41169
[junit4] 2> 844567 INFO (SUITE-MoveReplicaHDFSFailoverTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1010, maxMBSortInHeap=7.306786993210281, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2b2cf64c), locale=en-VG, timezone=America/North_Dakota/Center
[junit4] 2> NOTE: Linux 4.10.0-37-generic amd64/Oracle Corporation 10-ea (64-bit)/cpus=8,threads=2,free=184710816,total=493355008
[junit4] 2> NOTE: All tests run in this JVM: [TestTolerantUpdateProcessorCloud, TestDefaultStatsCache, TestSizeLimitedDistributedMap, TestInitQParser, TestLRUCache, TestEmbeddedSolrServerConstructors, HighlighterTest, TestExactStatsCache, CdcrBootstrapTest, ParsingFieldUpdateProcessorsTest, TestRealTimeGet, LegacyCloudClusterPropTest, TestShardHandlerFactory, TriLevelCompositeIdRoutingTest, TestReplicaProperties, FileBasedSpellCheckerTest, TestIndexSearcher, TestInPlaceUpdatesStandalone, TestManagedSchema, TestDynamicLoading, TestDeleteCollectionOnDownNodes, ChaosMonkeyNothingIsSafeTest, OverriddenZkACLAndCredentialsProvidersTest, TestFunctionQuery, ReplicaListTransformerTest, OverseerTest, StatsComponentTest, RequestLoggingTest, WordBreakSolrSpellCheckerTest, XmlUpdateRequestHandlerTest, TestCloudSchemaless, UniqFieldsUpdateProcessorFactoryTest, StatsReloadRaceTest, HttpPartitionTest, BooleanFieldTest, TestSolrXml, HttpTriggerListenerTest, DistributedExpandComponentTest, RecoveryZkTest, DistributedFacetPivotLongTailTest, TestCloudRecovery, BasicDistributedZk2Test, TestMultiValuedNumericRangeQuery, TestExpandComponent, AddSchemaFieldsUpdateProcessorFactoryTest, HdfsBasicDistributedZkTest, UnloadDistributedZkTest, TestReload, CopyFieldTest, TestCoreDiscovery, AliasIntegrationTest, TestTrie, ComputePlanActionTest, AsyncCallRequestStatusResponseTest, AnalyticsQueryTest, TestCloudNestedDocsSort, TestQuerySenderListener, DisMaxRequestHandlerTest, PluginInfoTest, TestSystemCollAutoCreate, TestSolrDeletionPolicy2, TestFilteredDocIdSet, TestExtendedDismaxParser, TestPartialUpdateDeduplication, AnalysisErrorHandlingTest, TestRTimerTree, TestRetrieveFieldsOptimizer, MoveReplicaHDFSFailoverTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MoveReplicaHDFSFailoverTest -Dtests.seed=F927525CAC20F3CB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-VG -Dtests.timezone=America/North_Dakota/Center -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J2 | MoveReplicaHDFSFailoverTest (suite) <<<
[junit4] > Throwable #1: java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
[junit4] > at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
[junit4] > at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
[junit4] > at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
[junit4] > at org.apache.solr.cloud.MoveReplicaHDFSFailoverTest.setupClass(MoveReplicaHDFSFailoverTest.java:55)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] Completed [248/750 (1!)] on J2 in 21.51s, 0 tests, 1 error <<< FAILURES!
[...truncated 84 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_F927525CAC20F3CB-001/init-core-data-001
[junit4] 2> 893816 WARN (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=19 numCloses=19
[junit4] 2> 893817 INFO (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 893817 INFO (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
[junit4] 2> 893818 INFO (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 893818 INFO (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest_F927525CAC20F3CB-001/tempDir-001
[junit4] 2> 893818 INFO (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 893819 INFO (Thread-1677) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 893819 INFO (Thread-1677) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 893821 ERROR (Thread-1677) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 893919 INFO (SUITE-HdfsAutoAddReplicasIntegrationTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:42203
[junit4] 2> 893928 INFO (jetty-launcher-3227-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 893928 INFO (jetty-launcher-3227-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 893928 INFO (jetty-launcher-3227-thread-3) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 893934 INFO (jetty-launcher-3227-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f4d1a74{/solr,null,AVAILABLE}
[junit4] 2> 893940 INFO (jetty-launcher-3227-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@ccd5b0{/solr,null,AVAILABLE}
[junit4] 2> 893940 INFO (jetty-launcher-3227-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3304756f{/solr,null,AVAILABLE}
[junit4] 2> 893945 INFO (jetty-launcher-3227-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@234c1c06{SSL,[ssl, http/1.1]}{127.0.0.1:42787}
[junit4] 2> 893945 INFO (jetty-launcher-3227-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@31dd048d{SSL,[ssl, http/1.1]}{127.0.0.1:43163}
[junit4] 2> 893945 INFO (jetty-launcher-3227-thread-3) [ ] o.e.j.s.Server Started @895459ms
[junit4] 2> 893945 INFO (jetty-launcher-3227-thread-2) [ ] o.e.j.s.Server Started @895459ms
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42787}
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43163}
[junit4] 2> 893945 INFO (jetty-launcher-3227-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3974ac7{SSL,[ssl, http/1.1]}{127.0.0.1:46073}
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-1) [ ] o.e.j.s.Server Started @895459ms
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46073}
[junit4] 2> 893946 ERROR (jetty-launcher-3227-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 893946 ERROR (jetty-launcher-3227-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 893946 ERROR (jetty-launcher-3227-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 893947 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T18:05:06.205012Z
[junit4] 2> 893947 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T18:05:06.205050Z
[junit4] 2> 893946 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 893947 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 893947 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 893947 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-11-20T18:05:06.205415Z
[junit4] 2> 893953 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 893954 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 893954 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 893962 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42203/solr
[junit4] 2> 893962 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42203/solr
[junit4] 2> 893964 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42203/solr
[junit4] 2> 893970 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15fda9ae89e0006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 893996 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 893996 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42787_solr
[junit4] 2> 893996 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.c.Overseer Overseer (id=99038075687927817-127.0.0.1:42787_solr-n_0000000000) starting
[junit4] 2> 894004 DEBUG (OverseerAutoScalingTriggerThread-99038075687927817-127.0.0.1:42787_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 894004 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42787_solr
[junit4] 2> 894007 INFO (zkCallback-3243-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 894008 DEBUG (OverseerAutoScalingTriggerThread-99038075687927817-127.0.0.1:42787_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 894008 DEBUG (OverseerAutoScalingTriggerThread-99038075687927817-127.0.0.1:42787_solr-n_0000000000) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:42787_solr]
[junit4] 2> 894008 DEBUG (OverseerAutoScalingTriggerThread-99038075687927817-127.0.0.1:42787_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 894008 DEBUG (OverseerAutoScalingTriggerThread-99038075687927817-127.0.0.1:42787_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 894009 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 894009 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 894010 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46073_solr
[junit4] 2> 894011 DEBUG (OverseerAutoScalingTriggerThread-99038075687927817-127.0.0.1:42787_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 894011 DEBUG (OverseerAutoScalingTriggerThread-99038075687927817-127.0.0.1:42787_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 894011 INFO (zkCallback-3244-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 894011 INFO (zkCallback-3243-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 894011 DEBUG (ScheduledTrigger-3425-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: [127.0.0.1:46073_solr, 127.0.0.1:42787_solr]
[junit4] 2> 894013 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 894014 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 894014 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43163_solr
[junit4] 2> 894015 INFO (zkCallback-3244-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 894015 INFO (zkCallback-3243-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 894015 INFO (zkCallback-3242-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 894049 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43163.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b99ecf6
[junit4] 2> 894052 INFO (jetty-launcher-3227-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46073.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b99ecf6
[junit4] 2> 894054 INFO (jetty-launcher-3227-thread-3) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42787.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b99ecf6
[junit4] 2> 894054 INFO (jetty-launcher-3227-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43163.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbea
[...truncated too long message...]
yTest-seed#[F927525CAC20F3CB]-worker) [ ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0
[junit4] 2> 2270725 INFO (SUITE-HdfsDirectoryTest-seed#[F927525CAC20F3CB]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Nov 20, 2017 6:28:02 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 64 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@bcbd55f), locale=ky-KG, timezone=SystemV/YST9
[junit4] 2> NOTE: Linux 4.10.0-37-generic amd64/Oracle Corporation 10-ea (64-bit)/cpus=8,threads=1,free=168541688,total=527433728
[junit4] 2> NOTE: All tests run in this JVM: [BlockDirectoryTest, TestCollectionAPIs, UninvertDocValuesMergePolicyTest, TestFastWriter, TestJmxIntegration, DebugComponentTest, TestStressCloudBlindAtomicUpdates, SolrGangliaReporterTest, TestSchemaNameResource, MoreLikeThisHandlerTest, TestSolrQueryParser, SpellingQueryConverterTest, TestSchemalessBufferedUpdates, TestTestInjection, TestMaxScoreQueryParser, HdfsChaosMonkeySafeLeaderTest, TestJsonFacetRefinement, DistributedVersionInfoTest, TestXIncludeConfig, MBeansHandlerTest, TestHashQParserPlugin, RemoteQueryErrorTest, MetricsHandlerTest, TestSQLHandler, TestOrdValues, HdfsBasicDistributedZk2Test, ShardRoutingTest, TestDFISimilarityFactory, PrimUtilsTest, HighlighterMaxOffsetTest, TestFieldCacheReopen, MissingSegmentRecoveryTest, CollectionTooManyReplicasTest, DeleteShardTest, DirectUpdateHandlerTest, SampleTest, NodeLostTriggerTest, TestAnalyzedSuggestions, TestHalfAndHalfDocValues, TestClusterStateMutator, RequestHandlersTest, TestFieldSortValues, SolrCmdDistributorTest, OpenExchangeRatesOrgProviderTest, VersionInfoTest, TestCloudInspectUtil, TestAuthorizationFramework, AddBlockUpdateTest, TestConfigSetProperties, SaslZkACLProviderTest, TestNumericTerms32, SignatureUpdateProcessorFactoryTest, CollectionsAPIDistributedZkTest, TestSolrFieldCacheBean, DistanceUnitsTest, TestReplicationHandlerBackup, OverseerTaskQueueTest, OverseerModifyCollectionTest, TestPseudoReturnFields, TestDistributedStatsComponentCardinality, ExitableDirectoryReaderTest, TestCustomStream, LargeFieldTest, AtomicUpdatesTest, CoreAdminRequestStatusTest, SegmentsInfoRequestHandlerTest, TestBlobHandler, DOMUtilTest, TestRestManager, DistributedFacetPivotLargeTest, TestExactSharedStatsCache, TestTrieFacet, LeaderFailoverAfterPartitionTest, SolrRequestParserTest, GraphQueryTest, TestLazyCores, BigEndianAscendingWordDeserializerTest, TestSubQueryTransformerDistrib, TestDocumentBuilder, ConcurrentDeleteAndCreateCollectionTest, TestRemoteStreaming, BinaryUpdateRequestHandlerTest, ZkCLITest, TestPivotHelperCode, SuggesterTSTTest, TestManagedSchemaThreadSafety, TestConfigSetsAPI, ChaosMonkeySafeLeaderWithPullReplicasTest, CurrencyFieldTypeTest, DeleteLastCustomShardedReplicaTest, RegexBoostProcessorTest, CdcrBidirectionalTest, TestScoreJoinQPNoScore, TestQuerySenderNoQuery, SolrMetricManagerTest, TestSmileRequest, SuggesterFSTTest, TestDynamicFieldCollectionResource, TestCryptoKeys, SolrXmlInZkTest, TestWriterPerf, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TimePartitionedUpdateProcessorTest, HdfsAutoAddReplicasIntegrationTest, ResponseHeaderTest, ClassificationUpdateProcessorTest, BadIndexSchemaTest, DocumentAnalysisRequestHandlerTest, TestRangeQuery, TlogReplayBufferedWhileIndexingTest, TestFaceting, TestSolrCloudWithDelegationTokens, HealthCheckHandlerTest, TestSchemaVersionResource, SolrIndexConfigTest, TestLegacyField, SolrIndexSplitterTest, TestSolrConfigHandler, TestClusterProperties, ConvertedLegacyTest, BlockJoinFacetDistribTest, ChaosMonkeySafeLeaderTest, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, BasicZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, ZkSolrClientTest, TestZkChroot, ShardRoutingCustomTest, TestRecovery, TestHashPartitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, DistributedTermsComponentTest, TestSolr4Spatial, SolrCoreTest, QueryElevationComponentTest, PeerSyncTest, TestSort, TestBadConfig, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SimplePostToolTest, CoreAdminHandlerTest, TestStressLucene, PolyFieldTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, DocValuesMultiTest, DistanceFunctionTest, TestSolrDeletionPolicy1, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, TestOmitPositions, TermsComponentTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, QueryParsingTest, PingRequestHandlerTest, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, TestLFUCache, HighlighterConfigTest, SOLR749Test, AlternateDirectoryTest, TestSolrIndexConfig, ResponseLogComponentTest, BadComponentTest, TestStressRecovery, MultiTermTest, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, TestElisionMultitermQuery, OutputWriterTest, TestSolrCoreProperties, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, PreAnalyzedFieldTest, ClusterStateTest, TestUtils, SystemInfoHandlerTest, UUIDFieldTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, CursorPagingTest, DistributedIntervalFacetingTest, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, AssignTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, DistribDocExpirationUpdateProcessorTest, LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, MultiThreadedOCPTest, ReplaceNodeTest, SSLMigrationTest, ShardSplitTest, AutoScalingHandlerTest, SearchRateTriggerTest, HdfsRecoverLeaseTest, HdfsRecoveryZkTest, ImplicitSnitchTest, BlobRepositoryCloudTest, CachingDirectoryFactoryTest, ConfigureRecoveryStrategyTest, CoreSorterTest, HdfsDirectoryFactoryTest, TestHdfsBackupRestoreCore, TestSolrConfigHandlerConcurrent, ShufflingReplicaListTransformerTest, SpatialHeatmapFacetsTest, CheckHdfsIndexTest, SolrSlf4jReporterTest, TestMacroExpander, TestMacros, JSONWriterTest, SmileWriterTest, TestBinaryResponseWriter, TestCustomDocTransformer, TestExportWriter, TestJavabinTupleStreamParser, TestPushWriter, TestRawResponseWriter, TestRawTransformer, TestSolrQueryResponse, TestChildDocTransformer, TestSubQueryTransformer, TestSubQueryTransformerCrossCore, TestManagedResource, TestBulkSchemaAPI, TestCopyFieldCollectionResource, TestDynamicFieldResource, TestFieldCollectionResource, TestFieldResource, TestSchemaResource, TestSchemaSimilarityResource, TestSerializedLuceneMatchVersion, TestUniqueKeyFieldResource, TestManagedStopFilterFactory, TestManagedSynonymFilterFactory, BadCopyFieldTest, ChangedSchemaMergeTest, DateRangeFieldTest, DocValuesMissingTest, EnumFieldTest, PreAnalyzedFieldManagedSchemaCloudTest, TestCloudManagedSchema, TestSchemaManager, TestUseDocValuesAsStored, CursorMarkTest, MergeStrategyTest, SortSpecParsingTest, TestAddFieldRealTimeGet, TestComplexPhraseLeadingWildcard, TestCustomSort, TestNoOpRegenerator, TestPayloadScoreQParserPlugin, TestRecoveryHdfs, BasicAuthIntegrationTest, TestPKIAuthenticationPlugin, TestSha256AuthenticationProvider, TestZkAclsWithHadoopAuth, TestAnalyzeInfixSuggestions, TestFileDictionaryLookup, BufferStoreTest, HdfsDirectoryTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=HdfsDirectoryTest -Dtests.seed=F927525CAC20F3CB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ky-KG -Dtests.timezone=SystemV/YST9 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | HdfsDirectoryTest (suite) <<<
[junit4] > Throwable #1: java.io.IOException: Timed out waiting for Mini HDFS Cluster to start
[junit4] > at __randomizedtesting.SeedInfo.seed([F927525CAC20F3CB]:0)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1207)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:840)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:746)
[junit4] > at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:616)
[junit4] > at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:105)
[junit4] > at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:63)
[junit4] > at org.apache.solr.store.hdfs.HdfsDirectoryTest.beforeClass(HdfsDirectoryTest.java:60)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] Completed [724/750 (11!)] on J1 in 14.74s, 0 tests, 1 error <<< FAILURES!
[...truncated 40181 lines...]