You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/10/28 12:32:48 UTC

[JENKINS-EA] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b85) - Build # 14698 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/14698/
Java: 32bit/jdk1.9.0-ea-b85 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.CdcrReplicationHandlerTest.doTest

Error Message:
Captured an uncaught exception in thread: Thread[id=10365, name=RecoveryThread-source_collection_shard1_replica1, state=RUNNABLE, group=TGRP-CdcrReplicationHandlerTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10365, name=RecoveryThread-source_collection_shard1_replica1, state=RUNNABLE, group=TGRP-CdcrReplicationHandlerTest]
	at __randomizedtesting.SeedInfo.seed([DECCE30FF25031D5:79885BAB9FEB226C]:0)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([DECCE30FF25031D5]:0)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:232)
Caused by: org.apache.solr.common.SolrException: java.io.FileNotFoundException: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationHandlerTest_DECCE30FF25031D5-001/jetty-001/cores/source_collection_shard1_replica1/data/tlog/tlog.0000000000000000007.1516273897924722688 (No such file or directory)
	at org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransactionLog.java:244)
	at org.apache.solr.update.CdcrTransactionLog.incref(CdcrTransactionLog.java:173)
	at org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1079)
	at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(UpdateLog.java:1579)
	at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(UpdateLog.java:1610)
	at org.apache.solr.core.SolrCore.seedVersionBuckets(SolrCore.java:877)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:534)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:225)
Caused by: java.io.FileNotFoundException: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationHandlerTest_DECCE30FF25031D5-001/jetty-001/cores/source_collection_shard1_replica1/data/tlog/tlog.0000000000000000007.1516273897924722688 (No such file or directory)
	at java.io.RandomAccessFile.open0(Native Method)
	at java.io.RandomAccessFile.open(RandomAccessFile.java:327)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
	at org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransactionLog.java:236)
	... 7 more


FAILED:  org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest

Error Message:
Server refused connection at: https://127.0.0.1:48833/nr/d

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: https://127.0.0.1:48833/nr/d
	at __randomizedtesting.SeedInfo.seed([DECCE30FF25031D5:13F278FCBDF14737]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:583)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest(DeleteInactiveReplicaTest.java:128)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:520)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1665)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:864)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:900)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:914)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:873)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:775)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:809)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:820)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:747)
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402)
	at java.net.Socket.connect(Socket.java:591)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:648)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:532)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:480)
	... 45 more




Build Log:
[...truncated 9534 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteInactiveReplicaTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/init-core-data-001
   [junit4]   2> 286319 INFO  (SUITE-DeleteInactiveReplicaTest-seed#[DECCE30FF25031D5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 286320 INFO  (SUITE-DeleteInactiveReplicaTest-seed#[DECCE30FF25031D5]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /nr/d
   [junit4]   2> 286321 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 286321 INFO  (Thread-856) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 286321 INFO  (Thread-856) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 286421 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.ZkTestServer start zk server on port:50037
   [junit4]   2> 286421 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 286422 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 286424 INFO  (zkCallback-214-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@eabb29 name:ZooKeeperConnection Watcher:127.0.0.1:50037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 286424 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 286425 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 286425 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 286427 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 286427 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 286431 INFO  (zkCallback-215-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@98246b name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 286431 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 286431 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 286431 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 286432 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 286434 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 286434 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 286435 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 286435 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 286436 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 286436 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 286437 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 286438 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 286438 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 286438 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 286439 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 286439 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 286440 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 286440 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 286441 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 286441 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 286442 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 286442 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 286443 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 286443 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 286444 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 286444 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 286446 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 286446 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 286451 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 286453 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 286454 INFO  (zkCallback-216-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c367f8 name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 286454 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 286454 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 286540 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1
   [junit4]   2> 286554 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 286556 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1847915{/nr/d,null,AVAILABLE}
   [junit4]   2> 286618 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.ServerConnector Started ServerConnector@1ff7127{SSL-http/1.1}{127.0.0.1:50233}
   [junit4]   2> 286618 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.Server Started @288211ms
   [junit4]   2> 286618 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/tempDir-001/control/data, hostContext=/nr/d, hostPort=50233, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores}
   [junit4]   2> 286618 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1c6c3b2
   [junit4]   2> 286618 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/'
   [junit4]   2> 286618 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 286618 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 286628 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 286628 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 286630 INFO  (zkCallback-217-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@14063d2 name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 286630 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 286631 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 286631 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 286631 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/solr.xml
   [junit4]   2> 286635 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores
   [junit4]   2> 286635 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoreContainer New CoreContainer 19680777
   [junit4]   2> 286635 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/]
   [junit4]   2> 286635 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoreContainer loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/lib
   [junit4]   2> 286635 WARN  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/lib).
   [junit4]   2> 286641 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 286643 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 286644 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 286644 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 286644 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50037/solr
   [junit4]   2> 286644 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 286645 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 286645 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 286646 INFO  (zkCallback-219-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3c2d7d name:ZooKeeperConnection Watcher:127.0.0.1:50037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 286646 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 286647 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 286648 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 286649 INFO  (zkCallback-220-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@17ceb51 name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 286650 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 286650 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 286651 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 286652 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 286653 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 286654 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 286655 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 286656 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 286656 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 286656 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 286657 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50233_nr%2Fd
   [junit4]   2> 286657 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:50233_nr%2Fd
   [junit4]   2> 286658 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 286659 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 286659 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 286659 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000
   [junit4]   2> 286659 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50233_nr%2Fd
   [junit4]   2> 286659 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 286660 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer Overseer (id=94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) starting
   [junit4]   2> 286661 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 286664 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 286664 INFO  (OverseerCollectionConfigSetProcessor-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 286664 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 286664 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 286697 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 286697 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 286697 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.CoresLocator Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores
   [junit4]   2> 286698 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 286698 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.CoresLocator Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/
   [junit4]   2> 286698 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 286699 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.s.SolrDispatchFilter user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 286699 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 286699 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 286699 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 286700 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:50233/nr/d",
   [junit4]   2>   "node_name":"127.0.0.1:50233_nr%2Fd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 286700 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 286700 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:50233/nr/d",
   [junit4]   2>   "node_name":"127.0.0.1:50233_nr%2Fd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 286700 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 286701 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 286701 INFO  (zkCallback-220-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 287700 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 287700 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 287700 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 287701 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 287701 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 287701 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/'
   [junit4]   2> 287701 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 287701 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 287708 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 287711 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 287713 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 287723 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 287723 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 287726 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 287773 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 287776 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 287776 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 287781 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 287783 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 287784 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 287784 WARN  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 287784 WARN  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 287784 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 287785 WARN  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 287785 WARN  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 287786 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 287786 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 287786 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 287786 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 287786 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79aea8
   [junit4]   2> 287786 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/data
   [junit4]   2> 287787 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/data/index/
   [junit4]   2> 287787 WARN  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 287787 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/data/index
   [junit4]   2> 287787 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=44, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 287787 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@bed97a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ffb7df),segFN=segments_1,generation=1}
   [junit4]   2> 287787 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 287788 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/data/
   [junit4]   2> 287788 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 287791 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 287793 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 287794 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 287795 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 287795 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 287797 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 287798 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 287798 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 287798 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 287799 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 287799 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 287799 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=31.630859375, floorSegmentMB=0.3955078125, forceMergeDeletesPctAllowed=28.90750504381536, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 287799 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@bed97a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ffb7df),segFN=segments_1,generation=1}
   [junit4]   2> 287799 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 287800 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@63ecc[collection1] main
   [junit4]   2> 287800 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 287800 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 287800 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 287800 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 287800 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 287800 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 287801 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 287801 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 287801 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 287801 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 287801 INFO  (searcherExecutor-968-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@63ecc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 287802 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 287802 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 287802 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 287802 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1516272847931047936
   [junit4]   2> 287804 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1516272847931047936
   [junit4]   2> 287804 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 287804 INFO  (coreLoadExecutor-967-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 287804 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:https://127.0.0.1:50233/nr/d collection:control_collection shard:shard1
   [junit4]   2> 287804 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 287805 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94767052898697221-core_node1-n_0000000000
   [junit4]   2> 287806 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 287806 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 287806 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 287806 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:50233/nr/d/collection1/
   [junit4]   2> 287806 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 287806 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 287806 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:50233/nr/d/collection1/ has no replicas
   [junit4]   2> 287806 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:50233/nr/d/collection1/ shard1
   [junit4]   2> 287807 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 287807 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 287808 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"https://127.0.0.1:50233/nr/d",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 287910 INFO  (zkCallback-220-thread-1-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 287959 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are https://127.0.0.1:50233/nr/d/collection1/ and leader is https://127.0.0.1:50233/nr/d/collection1/
   [junit4]   2> 287959 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50233/nr/d
   [junit4]   2> 287959 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 287959 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 287959 INFO  (coreZkRegister-961-thread-1-processing-n:127.0.0.1:50233_nr%2Fd x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50233_nr%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 287960 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:50233/nr/d",
   [junit4]   2>   "node_name":"127.0.0.1:50233_nr%2Fd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 287960 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:50233/nr/d",
   [junit4]   2>   "node_name":"127.0.0.1:50233_nr%2Fd",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 288061 INFO  (zkCallback-220-thread-2-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 288202 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 288202 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 288204 INFO  (zkCallback-222-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@462c81 name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 288204 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 288204 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 288204 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 288205 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 288205 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 288205 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 288206 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 288206 INFO  (zkCallback-223-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1e32121 name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 288206 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 288207 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 288208 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 288208 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 288258 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/cores/collection1
   [junit4]   2> 288259 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001
   [junit4]   2> 288260 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 288260 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f50a5{/nr/d,null,AVAILABLE}
   [junit4]   2> 288268 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.ServerConnector Started ServerConnector@1610159{SSL-http/1.1}{127.0.0.1:42559}
   [junit4]   2> 288268 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.e.j.s.Server Started @289861ms
   [junit4]   2> 288268 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/nr/d, hostPort=42559, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/cores}
   [junit4]   2> 288269 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@1c6c3b2
   [junit4]   2> 288269 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/'
   [junit4]   2> 288269 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 288269 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 288279 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 288279 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 288281 INFO  (zkCallback-224-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1d63113 name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 288281 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 288282 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 288283 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 288283 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/solr.xml
   [junit4]   2> 288287 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/cores
   [junit4]   2> 288287 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoreContainer New CoreContainer 28882611
   [junit4]   2> 288287 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/]
   [junit4]   2> 288287 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.CoreContainer loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/lib
   [junit4]   2> 288287 WARN  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/lib).
   [junit4]   2> 288297 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 288299 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 288300 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 288300 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 288300 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50037/solr
   [junit4]   2> 288300 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 288300 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 288301 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 288302 INFO  (zkCallback-226-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1f1332e name:ZooKeeperConnection Watcher:127.0.0.1:50037 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 288302 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 288303 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 288304 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 288305 INFO  (zkCallback-227-thread-1-processing-n:127.0.0.1:42559_nr%2Fd) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@f37276 name:ZooKeeperConnection Watcher:127.0.0.1:50037/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 288305 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 288307 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 288310 INFO  (OverseerStateUpdate-94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 289309 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42559_nr%2Fd
   [junit4]   2> 289309 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:42559_nr%2Fd
   [junit4]   2> 289310 INFO  (zkCallback-222-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 289310 INFO  (zkCallback-227-thread-1-processing-n:127.0.0.1:42559_nr%2Fd) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 289310 INFO  (zkCallback-220-thread-2-processing-n:127.0.0.1:50233_nr%2Fd) [n:127.0.0.1:50233_nr%2Fd    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 289311 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 289311 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94767052898697226-127.0.0.1:42559_nr%2Fd-n_0000000001
   [junit4]   2> 289312 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94767052898697221-127.0.0.1:50233_nr%2Fd-n_0000000000 to know if I could be the leader
   [junit4]   2> 289325 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 289326 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 289326 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.CoresLocator Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/cores
   [junit4]   2> 289326 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 289326 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.CoresLocator Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteInactiveReplicaTest_DECCE30FF25031D5-001/shard-1-001/cores/collection1/
   [junit4]   2> 289327 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveReplicaTest-seed#[DECCE30FF25031D5]) [n:127.0.0.1:42559_nr%2Fd    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 289327 INFO  (TEST-DeleteInactiveReplicaTest.deleteInactiveRepli

[...truncated too long message...]

	/solr/clusterstate.json
   [junit4]   2> 	15	/solr/collections/source_collection/cdcr/state/buffer
   [junit4]   2> 	15	/solr/collections/source_collection/cdcr/state/process
   [junit4]   2> 	15	/solr/collections/source_collection/state.json
   [junit4]   2> 	5	/solr/collections/source_collection/leader_elect/shard1/election/94767114909712392-core_node1-n_0000000000
   [junit4]   2> 	3	/solr/overseer_elect/election/94767114909712392-127.0.0.1:54623_-n_0000000001
   [junit4]   2> 	3	/solr/collections/source_collection/leader_elect/shard1/election/94767114909712392-core_node2-n_0000000000
   [junit4]   2> 	3	/solr/collections/source_collection/leader_elect/shard1/election/94767114909712459-core_node1-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/94767114909712459-127.0.0.1:46021_-n_0000000004
   [junit4]   2> 	2	/solr/collections/tmp_collection/cdcr/state/buffer
   [junit4]   2> 	2	/solr/collections/tmp_collection/cdcr/state/process
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	91	/solr/collections
   [junit4]   2> 	86	/solr/live_nodes
   [junit4]   2> 	4	/solr/overseer/collection-queue-work
   [junit4]   2> 	3	/solr/overseer/queue
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CdcrReplicationHandlerTest -Dtests.method=doTest -Dtests.seed=DECCE30FF25031D5 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_PS -Dtests.timezone=Canada/Pacific -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    362s J0 | CdcrReplicationHandlerTest.doTest <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10365, name=RecoveryThread-source_collection_shard1_replica1, state=RUNNABLE, group=TGRP-CdcrReplicationHandlerTest]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DECCE30FF25031D5:79885BAB9FEB226C]:0)
   [junit4]    > Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DECCE30FF25031D5]:0)
   [junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:232)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: java.io.FileNotFoundException: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationHandlerTest_DECCE30FF25031D5-001/jetty-001/cores/source_collection_shard1_replica1/data/tlog/tlog.0000000000000000007.1516273897924722688 (No such file or directory)
   [junit4]    > 	at org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransactionLog.java:244)
   [junit4]    > 	at org.apache.solr.update.CdcrTransactionLog.incref(CdcrTransactionLog.java:173)
   [junit4]    > 	at org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1079)
   [junit4]    > 	at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(UpdateLog.java:1579)
   [junit4]    > 	at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(UpdateLog.java:1610)
   [junit4]    > 	at org.apache.solr.core.SolrCore.seedVersionBuckets(SolrCore.java:877)
   [junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:534)
   [junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:225)
   [junit4]    > Caused by: java.io.FileNotFoundException: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationHandlerTest_DECCE30FF25031D5-001/jetty-001/cores/source_collection_shard1_replica1/data/tlog/tlog.0000000000000000007.1516273897924722688 (No such file or directory)
   [junit4]    > 	at java.io.RandomAccessFile.open0(Native Method)
   [junit4]    > 	at java.io.RandomAccessFile.open(RandomAccessFile.java:327)
   [junit4]    > 	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
   [junit4]    > 	at org.apache.solr.update.CdcrTransactionLog.reopenOutputStream(CdcrTransactionLog.java:236)
   [junit4]    > 	... 7 more
   [junit4]   2> 1594318 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[DECCE30FF25031D5]-worker) [n:127.0.0.1:54623_ c:source_collection s:shard1 r:core_node2 x:source_collection_shard1_replica1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Oct 28, 2015 11:29:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {_version_=BlockTreeOrds(blocksize=128), id=FST50}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ar_PS, timezone=Canada/Pacific
   [junit4]   2> NOTE: Linux 3.19.0-31-generic i386/Oracle Corporation 1.9.0-ea (32-bit)/cpus=12,threads=1,free=80176064,total=450625536
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues, AnalysisErrorHandlingTest, IndexSchemaRuntimeFieldTest, TestAnalyzedSuggestions, TestFilteredDocIdSet, TestSchemaResource, TestExceedMaxTermLength, JSONWriterTest, TestRandomDVFaceting, TestMissingGroups, SmileWriterTest, OverseerTaskQueueTest, XsltUpdateRequestHandlerTest, ExternalFileFieldSortTest, TestRangeQuery, DateRangeFieldTest, TestDistributedMissingSort, QueryEqualityTest, TestReplicaProperties, TestMergePolicyConfig, DistribJoinFromCollectionTest, HdfsUnloadDistributedZkTest, TestIBSimilarityFactory, OverseerRolesTest, TestConfig, TestConfigSetsAPIZkFailure, RankQueryTest, TestSolr4Spatial2, LeaderInitiatedRecoveryOnCommitTest, TestRecoveryHdfs, DateFieldTest, StandardRequestHandlerTest, HdfsChaosMonkeyNothingIsSafeTest, TestPivotHelperCode, DistributedVersionInfoTest, TestSimpleQParserPlugin, DeleteInactiveReplicaTest, TestValueSourceCache, TestTrackingShardHandlerFactory, AliasIntegrationTest, TestHighlightDedupGrouping, LeaderElectionIntegrationTest, TestSolr4Spatial, SignatureUpdateProcessorFactoryTest, TestDocSet, TestFunctionQuery, DisMaxRequestHandlerTest, TestSolrDynamicMBean, AssignTest, TestDynamicFieldResource, SpellingQueryConverterTest, EnumFieldTest, TestRequestStatusCollectionAPI, ReplicationFactorTest, TestIntervalFaceting, PolyFieldTest, TestSolrXml, SolrRequestParserTest, TestFaceting, SimpleCollectionCreateDeleteTest, TestRTimerTree, FileUtilsTest, PrimUtilsTest, SolrCoreTest, SyncSliceTest, ResourceLoaderTest, TestSolrJ, ExitableDirectoryReaderTest, HdfsChaosMonkeySafeLeaderTest, TestCloudManagedSchemaConcurrent, BadCopyFieldTest, TestBlobHandler, FileBasedSpellCheckerTest, TestAtomicUpdateErrorCases, BasicZkTest, HdfsThreadLeakTest, HttpPartitionTest, TestSolrConfigHandler, TestInitParams, TestBinaryField, RemoteQueryErrorTest, CollectionStateFormat2Test, TestRestManager, CoreMergeIndexesAdminHandlerTest, TestLuceneMatchVersion, TestNoOpRegenerator, BigEndianAscendingWordSerializerTest, TestChildDocTransformer, TestDefaultStatsCache, TestHighFrequencyDictionaryFactory, TestManagedSchemaFieldResource, TestReplicationHandler, TestHashPartitioner, TermVectorComponentDistributedTest, TestRealTimeGet, TestJoin, StatsComponentTest, TestGroupingSearch, QueryElevationComponentTest, TestSort, BasicFunctionalityTest, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SpellCheckCollatorTest, TestStressLucene, WordBreakSolrSpellCheckerTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, DistanceFunctionTest, IndexBasedSpellCheckerTest, TestQueryTypes, RequestHandlersTest, TestIndexingPerformance, TestArbitraryIndexDir, UniqFieldsUpdateProcessorFactoryTest, TestComponentsName, TestBinaryResponseWriter, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, UpdateParamsTest, AlternateDirectoryTest, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, SampleTest, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, TestPhraseSuggestions, TestCharFilters, SynonymTokenizerTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, TimeZoneUtilsTest, OpenExchangeRatesOrgProviderTest, PluginInfoTest, DateMathParserTest, DOMUtilTest, TestUtils, SliceStateTest, CursorPagingTest, SolrTestCaseJ4Test, TestSimpleTrackingShardHandler, TestTolerantSearch, TestEmbeddedSolrServerConstructors, ActionThrottleTest, AsyncCallRequestStatusResponseTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest]
   [junit4] Completed [447/553] on J0 in 362.59s, 1 test, 1 error <<< FAILURES!

[...truncated 341 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:775: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:719: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:59: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:233: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:516: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1432: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:987: There were test failures: 553 suites (8 ignored), 2175 tests, 2 errors, 76 ignored (38 assumptions) [seed: DECCE30FF25031D5]

Total time: 60 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any