You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/03/24 15:47:54 UTC

[JENKINS] Lucene-Solr-7.x-Windows (32bit/jdk1.8.0_144) - Build # 517 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/517/
Java: 32bit/jdk1.8.0_144 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestCloudRecovery

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.     C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.     C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.     C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.     C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.

   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.

   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.

   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.

   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001
   C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001

	at __randomizedtesting.SeedInfo.seed([7F56965C8F1687C1]:0)
	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:318)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1798 lines...]
   [junit4] JVM J1: stdout was not empty, see: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\build\core\test\temp\junit4-J1-20180324_135943_7227569650528619132289.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] codec: HighCompressionCompressingStoredFields, pf: TestBloomFilteredLucenePostings, dvf: Lucene70
   [junit4] <<< JVM J1: EOF ----

[...truncated 10987 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\init-core-data-001
   [junit4]   2> 786823 WARN  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 786823 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 786828 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 786828 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 786830 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001
   [junit4]   2> 786830 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 786830 INFO  (Thread-1903) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 786830 INFO  (Thread-1903) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 786834 ERROR (Thread-1903) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 786930 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:59173
   [junit4]   2> 786934 INFO  (zkConnectionManagerCallback-1604-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 786940 INFO  (jetty-launcher-1601-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T09:27:37-12:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 786940 INFO  (jetty-launcher-1601-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 786940 INFO  (jetty-launcher-1601-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 786940 INFO  (jetty-launcher-1601-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 786940 INFO  (jetty-launcher-1601-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@9d63f2{/solr,null,AVAILABLE}
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T09:27:37-12:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@10ba9f2{HTTP/1.1,[http/1.1]}{127.0.0.1:59177}
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-1) [    ] o.e.j.s.Server Started @795044ms
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59177}
   [junit4]   2> 786942 ERROR (jetty-launcher-1601-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@a8aae8{/solr,null,AVAILABLE}
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-24T14:32:53.255Z
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@1c82ac{HTTP/1.1,[http/1.1]}{127.0.0.1:59180}
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.e.j.s.Server Started @795045ms
   [junit4]   2> 786942 INFO  (jetty-launcher-1601-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59180}
   [junit4]   2> 786943 ERROR (jetty-launcher-1601-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 786943 INFO  (jetty-launcher-1601-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 786943 INFO  (jetty-launcher-1601-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 786943 INFO  (jetty-launcher-1601-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 786943 INFO  (jetty-launcher-1601-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-24T14:32:53.256Z
   [junit4]   2> 786960 INFO  (zkConnectionManagerCallback-1606-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 786961 INFO  (zkConnectionManagerCallback-1608-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 786961 INFO  (jetty-launcher-1601-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 786962 INFO  (jetty-launcher-1601-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 786970 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10000a589180002, likely client has closed socket
   [junit4]   2> 786977 INFO  (jetty-launcher-1601-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59173/solr
   [junit4]   2> 786977 INFO  (jetty-launcher-1601-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59173/solr
   [junit4]   2> 786980 INFO  (zkConnectionManagerCallback-1616-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 786980 INFO  (zkConnectionManagerCallback-1614-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 786980 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10000a589180004, likely client has closed socket
   [junit4]   2> 786983 INFO  (zkConnectionManagerCallback-1620-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 786983 INFO  (zkConnectionManagerCallback-1618-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 787018 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 787018 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59177_solr
   [junit4]   2> 787018 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.c.Overseer Overseer (id=72058305007583237-127.0.0.1:59177_solr-n_0000000000) starting
   [junit4]   2> 787026 INFO  (zkConnectionManagerCallback-1626-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 787030 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59173/solr ready
   [junit4]   2> 787031 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59177_solr
   [junit4]   2> 787045 INFO  (OverseerStateUpdate-72058305007583237-127.0.0.1:59177_solr-n_0000000000) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 787097 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 787098 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 787100 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59180_solr
   [junit4]   2> 787393 INFO  (zkCallback-1619-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 787395 INFO  (zkCallback-1625-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 787395 INFO  (zkCallback-1617-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 787547 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59180.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 787554 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59180.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 787555 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59180.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 787557 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\.
   [junit4]   2> 787562 INFO  (zkConnectionManagerCallback-1630-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 787564 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 787564 INFO  (jetty-launcher-1601-thread-2) [n:127.0.0.1:59180_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59173/solr ready
   [junit4]   2> 788166 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59177.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 788174 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59177.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 788177 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59177.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 788178 INFO  (jetty-launcher-1601-thread-1) [n:127.0.0.1:59177_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\.
   [junit4]   2> 788222 INFO  (zkConnectionManagerCallback-1632-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 788235 INFO  (zkConnectionManagerCallback-1636-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 788243 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 788244 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59173/solr ready
   [junit4]   2> 788253 INFO  (qtp10022953-7120) [n:127.0.0.1:59177_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=2&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=2&action=CREATE&numShards=2&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 788255 INFO  (OverseerThreadFactory-3117-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 788374 INFO  (OverseerStateUpdate-72058305007583237-127.0.0.1:59177_solr-n_0000000000) [n:127.0.0.1:59177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:59177/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 788379 INFO  (OverseerStateUpdate-72058305007583237-127.0.0.1:59177_solr-n_0000000000) [n:127.0.0.1:59177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:59180/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 788381 INFO  (OverseerStateUpdate-72058305007583237-127.0.0.1:59177_solr-n_0000000000) [n:127.0.0.1:59177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"collection1_shard2_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:59177/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 788383 INFO  (OverseerStateUpdate-72058305007583237-127.0.0.1:59177_solr-n_0000000000) [n:127.0.0.1:59177_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "core":"collection1_shard2_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:59180/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 788591 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_n2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 788592 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 788593 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 788593 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 788594 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_n4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 788594 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_n6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 788710 INFO  (zkCallback-1619-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 788710 INFO  (zkCallback-1617-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 788710 INFO  (zkCallback-1617-thread-2-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 788715 INFO  (zkCallback-1619-thread-2-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 789636 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 789637 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 789639 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 789656 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.s.IndexSchema [collection1_shard2_replica_n6] Schema name=minimal
   [junit4]   2> 789659 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema [collection1_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 789660 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 789660 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 789660 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n2' using configuration from collection collection1, trusted=true
   [junit4]   2> 789660 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n6' using configuration from collection collection1, trusted=true
   [junit4]   2> 789661 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59180.solr.core.collection1.shard1.replica_n2' (registry 'solr.core.collection1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 789661 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59180.solr.core.collection1.shard2.replica_n6' (registry 'solr.core.collection1.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 789661 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 789661 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 789661 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.SolrCore [[collection1_shard2_replica_n6] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\.\collection1_shard2_replica_n6\data\]
   [junit4]   2> 789662 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 789665 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.s.IndexSchema [collection1_shard2_replica_n4] Schema name=minimal
   [junit4]   2> 789667 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 789667 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n4' using configuration from collection collection1, trusted=true
   [junit4]   2> 789668 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59177.solr.core.collection1.shard2.replica_n4' (registry 'solr.core.collection1.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 789668 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 789668 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.SolrCore [[collection1_shard2_replica_n4] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\.\collection1_shard2_replica_n4\data\]
   [junit4]   2> 789662 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [[collection1_shard1_replica_n2] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\.\collection1_shard1_replica_n2\data\]
   [junit4]   2> 789684 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 789686 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 789687 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
   [junit4]   2> 789687 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59177.solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1eaff6b
   [junit4]   2> 789687 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 789688 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\.\collection1_shard1_replica_n1\data\]
   [junit4]   2> 790372 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 790372 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 790375 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 790375 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 790380 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@1dd563d[collection1_shard2_replica_n4] main]
   [junit4]   2> 790381 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 790381 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 790381 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 790381 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 790383 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 790383 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 790384 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 790385 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 790386 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@48f1ce[collection1_shard1_replica_n1] main]
   [junit4]   2> 790387 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 790387 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 790388 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 790388 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 790388 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 790389 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 790389 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595829887122276352
   [junit4]   2> 790390 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@e63f48[collection1_shard2_replica_n6] main]
   [junit4]   2> 790390 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 790391 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 790392 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595829887125422080
   [junit4]   2> 790393 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 790393 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 790394 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 790394 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 790395 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 790395 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595829887128567808
   [junit4]   2> 790397 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node7=0}, version=0}
   [junit4]   2> 790398 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@2e5d7a[collection1_shard1_replica_n2] main]
   [junit4]   2> 790400 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
   [junit4]   2> 790400 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
   [junit4]   2> 790401 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=0}, version=0}
   [junit4]   2> 790401 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node7=0, core_node8=0}, version=1}
   [junit4]   2> 790405 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 790405 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1595829887139053568
   [junit4]   2> 790406 INFO  (searcherExecutor-3123-thread-1-processing-n:127.0.0.1:59180_solr x:collection1_shard2_replica_n6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.SolrCore [collection1_shard2_replica_n6] Registered new searcher Searcher@e63f48[collection1_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 790406 INFO  (searcherExecutor-3124-thread-1-processing-n:127.0.0.1:59177_solr x:collection1_shard2_replica_n4 s:shard2 c:collection1 r:core_node7) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.SolrCore [collection1_shard2_replica_n4] Registered new searcher Searcher@1dd563d[collection1_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 790407 INFO  (searcherExecutor-3125-thread-1-processing-n:127.0.0.1:59177_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@48f1ce[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 790408 INFO  (searcherExecutor-3122-thread-1-processing-n:127.0.0.1:59180_solr x:collection1_shard1_replica_n2 s:shard1 c:collection1 r:core_node5) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [collection1_shard1_replica_n2] Registered new searcher Searcher@2e5d7a[collection1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 790408 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 790409 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 790409 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/
   [junit4]   2> 790410 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_n6 url=http://127.0.0.1:59180/solr START replicas=[http://127.0.0.1:59177/solr/collection1_shard2_replica_n4/] nUpdates=100
   [junit4]   2> 790410 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
   [junit4]   2> 790412 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 790413 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.S.Request [collection1_shard2_replica_n4]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 790517 INFO  (zkCallback-1619-thread-2-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790517 INFO  (zkCallback-1619-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790517 INFO  (zkCallback-1617-thread-2-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790517 INFO  (zkCallback-1617-thread-3-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790711 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_n6 url=http://127.0.0.1:59180/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 790712 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 790712 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 790712 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 790717 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/ shard2
   [junit4]   2> 790818 INFO  (zkCallback-1617-thread-3-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790818 INFO  (zkCallback-1619-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790818 INFO  (zkCallback-1617-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790818 INFO  (zkCallback-1619-thread-2-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790819 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 790821 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_n6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2228
   [junit4]   2> 790913 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 790913 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 790913 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/
   [junit4]   2> 790913 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_n1 url=http://127.0.0.1:59177/solr START replicas=[http://127.0.0.1:59180/solr/collection1_shard1_replica_n2/] nUpdates=100
   [junit4]   2> 790946 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.c.S.Request [collection1_shard1_replica_n2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=36
   [junit4]   2> 790956 INFO  (zkCallback-1619-thread-2-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790956 INFO  (zkCallback-1619-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790956 INFO  (zkCallback-1617-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 790956 INFO  (zkCallback-1617-thread-3-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791213 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_n1 url=http://127.0.0.1:59177/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 791213 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 791213 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 791213 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 791213 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/ shard1
   [junit4]   2> 791336 INFO  (zkCallback-1617-thread-3-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791336 INFO  (zkCallback-1617-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791337 INFO  (zkCallback-1619-thread-2-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791337 INFO  (zkCallback-1619-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791341 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 791346 INFO  (qtp10022953-7142) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2753
   [junit4]   2> 791407 INFO  (qtp10022953-7143) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_n4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2817
   [junit4]   2> 791423 INFO  (qtp113854-7129) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_n2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2834
   [junit4]   2> 791438 INFO  (qtp10022953-7120) [n:127.0.0.1:59177_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 791535 INFO  (zkCallback-1619-thread-2-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791535 INFO  (zkCallback-1619-thread-1-processing-n:127.0.0.1:59180_solr) [n:127.0.0.1:59180_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791535 INFO  (zkCallback-1617-thread-1-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 791535 INFO  (zkCallback-1617-thread-2-processing-n:127.0.0.1:59177_solr) [n:127.0.0.1:59177_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 792338 INFO  (OverseerCollectionConfigSetProcessor-72058305007583237-127.0.0.1:59177_solr-n_0000000000) [n:127.0.0.1:59177_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 792438 INFO  (qtp10022953-7120) [n:127.0.0.1:59177_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=2&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=2&action=CREATE&numShards=2&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=4184
   [junit4]   2> 792438 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> 792439 INFO  (SUITE-TestCloudRecovery-seed#[7F56965C8F1687C1]-worker) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
   [junit4]   2> 792463 INFO  (TEST-TestCloudRecovery.corruptedLogTest-seed#[7F56965C8F1687C1]) [    ] o.a.s.SolrTestCaseJ4 ###Starting corruptedLogTest
   [junit4]   2> 792503 INFO  (qtp10022953-7126) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1595829889323237376&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{deleteByQuery=*:* (-1595829889323237376)} 0 9
   [junit4]   2> 792519 INFO  (qtp113854-7132) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1595829889325334528&distrib.from=http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1595829889325334528)} 0 21
   [junit4]   2> 792520 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=1, core_node5=1}, version=2}
   [junit4]   2> 792520 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{deleteByQuery=*:* (-1595829889325334528)} 0 30
   [junit4]   2> 792526 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node7=1, core_node8=1}, version=2}
   [junit4]   2> 792526 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1595829889323237376)} 0 37
   [junit4]   2> 792580 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595829889419706368,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 792580 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 792582 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 792582 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
   [junit4]   2> 792584 INFO  (qtp113854-7132) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595829889423900672,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 792585 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595829889424949248,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 792585 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 792585 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 792586 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
   [junit4]   2> 792604 INFO  (qtp113854-7132) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 792627 INFO  (qtp113854-7132) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 792627 INFO  (qtp113854-7132) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 42
   [junit4]   2> 792629 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1595829889471086592,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 792629 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 792630 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 792630 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 24
   [junit4]   2> 792631 INFO  (qtp113854-7136) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={_stateVer_=collection1:7&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 104
   [junit4]   2> 792655 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{add=[1000 (1595829889475280896)]} 0 18
   [junit4]   2> 792656 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1000 (1595829889475280896)]} 0 22
   [junit4]   2> 792684 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{add=[1001 (1595829889499398144)]} 0 23
   [junit4]   2> 792687 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1001 (1595829889499398144)]} 0 32
   [junit4]   2> 792691 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{add=[0 (1595829889531904000)]} 0 0
   [junit4]   2> 792691 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0 (1595829889531904000)]} 0 5
   [junit4]   2> 792736 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{add=[1 (1595829889536098304)]} 0 42
   [junit4]   2> 792743 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1595829889536098304)]} 0 47
   [junit4]   2> 792788 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{add=[2 (1595829889599012864)]} 0 0
   [junit4]   2> 792790 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2 (1595829889599012864)]} 0 35
   [junit4]   2> 792791 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{add=[3 (1595829889640955904)]} 0 0
   [junit4]   2> 792792 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3 (1595829889640955904)]} 0 1
   [junit4]   2> 792804 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{add=[4 (1595829889644101632)]} 0 6
   [junit4]   2> 792804 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4 (1595829889644101632)]} 0 9
   [junit4]   2> 792806 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{add=[5 (1595829889656684544)]} 0 0
   [junit4]   2> 792806 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5 (1595829889656684544)]} 0 1
   [junit4]   2> 792808 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{add=[6 (1595829889657733120)]} 0 0
   [junit4]   2> 792808 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6 (1595829889657733120)]} 0 1
   [junit4]   2> 792809 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{add=[7 (1595829889659830272)]} 0 0
   [junit4]   2> 792811 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7 (1595829889659830272)]} 0 1
   [junit4]   2> 792835 INFO  (qtp113854-7133) [n:127.0.0.1:59180_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59177/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{add=[8 (1595829889662976000)]} 0 22
   [junit4]   2> 792835 INFO  (qtp10022953-7127) [n:127.0.0.1:59177_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8 (1595829889662976000)]} 0 23
   [junit4]   2> 792851 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n4]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:59180/solr/collection1_shard2_replica_n6/&wt=javabin&version=2}{add=[9 (1595829889688141824)]} 0 14
   [junit4]   2> 792851 INFO  (qtp113854-7144) [n:127.0.0.1:59180_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_n6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n6]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9 (1595829889688141824)]} 0 15
   [junit4]   2> 792853 INFO  (qtp10022953-7123) [n:127.0.0.1:59177_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_n4] o.a.s.c.S.Request [collection1_shard2_replica_n4]  webapp=/solr path=/sel

[...truncated too long message...]

0000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data\tlog
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2\data
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard1_replica_n2
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data\tlog
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6\data
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2\collection1_shard2_replica_n6
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node2
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data\tlog
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1\data
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard1_replica_n1
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001: java.nio.file.FileSystemException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog\tlog.0000000000000000001: The process cannot access the file because it is being used by another process.
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data\tlog
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4\data
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1\collection1_shard2_replica_n4
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001\node1
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001\tempDir-001
   [junit4]    >    C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestCloudRecovery_7F56965C8F1687C1-001
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7F56965C8F1687C1]:0)
   [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:318)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4] Completed [191/784 (1!)] on J1 in 34.72s, 2 tests, 1 error <<< FAILURES!

[...truncated 47282 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/517/consoleText

[repro] Revision: bf33c0a4c0d518a2afd4e06acff06931a61ef16c

[repro] Ant options: "-Dargs=-server -XX:+UseParallelGC"

[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr\core
[repro]       TestCloudRecovery
[repro] ant compile-test

[...truncated 3335 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestCloudRecovery" -Dtests.showOutput=onerror "-Dargs=-server -XX:+UseParallelGC" -Dtests.seed=7F56965C8F1687C1 -Dtests.slow=true -Dtests.locale=sk -Dtests.timezone=Europe/Bucharest -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 74 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.TestCloudRecovery
[repro] Exiting with code 0

[...truncated 75 lines...]

[JENKINS] Lucene-Solr-7.x-Windows (32bit/jdk1.8.0_144) - Build # 518 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/518/
Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC

14 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:61989/_/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:61989/_/collection1
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5:9092946DA6AF385D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1105)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1591)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:212)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 53 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=295, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61893), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)    2) Thread[id=294, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    3) Thread[id=296, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    4) Thread[id=297, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 
   1) Thread[id=295, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61893), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
   2) Thread[id=294, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   3) Thread[id=296, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   4) Thread[id=297, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=295, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61893), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)    2) Thread[id=297, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=295, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61893), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
   2) Thread[id=297, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:61995/_/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:61995/_/collection1
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5:9092946DA6AF385D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1105)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1591)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:212)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 53 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=294, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61917), state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)    2) Thread[id=295, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    3) Thread[id=296, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=293, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 
   1) Thread[id=294, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61917), state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
   2) Thread[id=295, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   3) Thread[id=296, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=293, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=294, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61917), state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)    2) Thread[id=296, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=294, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:61917), state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
   2) Thread[id=296, name=zkConnectionManagerCallback-91-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:53064/_/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:53064/_/collection1
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5:9092946DA6AF385D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1105)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1591)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:212)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
	... 53 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:     1) Thread[id=18339, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=18341, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)    3) Thread[id=18342, name=zkConnectionManagerCallback-4015-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    4) Thread[id=18340, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:53005), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1249)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1173)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 
   1) Thread[id=18339, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=18341, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
   3) Thread[id=18342, name=zkConnectionManagerCallback-4015-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   4) Thread[id=18340, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:53005), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1249)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1173)
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=18342, name=zkConnectionManagerCallback-4015-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:748)    2) Thread[id=18340, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:53005), state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=18342, name=zkConnectionManagerCallback-4015-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   2) Thread[id=18340, name=TEST-ChaosMonkeyNothingIsSafeTest.test-seed#[18C6ABB7085355A5]-SendThread(127.0.0.1:53005), state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5]:0)


FAILED:  org.apache.solr.cloud.CreateRoutedAliasTest.testV1

Error Message:
Unexpected status: HTTP/1.1 400 Bad Request

Stack Trace:
java.lang.AssertionError: Unexpected status: HTTP/1.1 400 Bad Request
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5:B7086CC749BEF935]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.CreateRoutedAliasTest.assertSuccess(CreateRoutedAliasTest.java:366)
	at org.apache.solr.cloud.CreateRoutedAliasTest.testV1(CreateRoutedAliasTest.java:198)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CreateRoutedAliasTest.testTimezoneAbsoluteDate

Error Message:
Error from server at http://127.0.0.1:62734/solr: Can't create collection alias for collections='testTimezoneAbsoluteDate_2018-01-15', 'testTimezoneAbsoluteDate_2018-01-15' is not an existing collection or alias

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:62734/solr: Can't create collection alias for collections='testTimezoneAbsoluteDate_2018-01-15', 'testTimezoneAbsoluteDate_2018-01-15' is not an existing collection or alias
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5:BD26D6DA9C140CDB]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1105)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:885)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:818)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
	at org.apache.solr.cloud.CreateRoutedAliasTest.testTimezoneAbsoluteDate(CreateRoutedAliasTest.java:237)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CreateRoutedAliasTest.testV2

Error Message:
Unexpected status: HTTP/1.1 400 Bad Request

Stack Trace:
java.lang.AssertionError: Unexpected status: HTTP/1.1 400 Bad Request
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5:D983F7D798B8F074]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.CreateRoutedAliasTest.assertSuccess(CreateRoutedAliasTest.java:366)
	at org.apache.solr.cloud.CreateRoutedAliasTest.testV2(CreateRoutedAliasTest.java:140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.CreateRoutedAliasTest.testCollectionNamesMustBeAbsent

Error Message:
Expected collection1 to be created with 2 shards and 1 replica null Live Nodes: [127.0.0.1:62734_solr, 127.0.0.1:62737_solr] Last available state: null

Stack Trace:
java.lang.AssertionError: Expected collection1 to be created with 2 shards and 1 replica
null
Live Nodes: [127.0.0.1:62734_solr, 127.0.0.1:62737_solr]
Last available state: null
	at __randomizedtesting.SeedInfo.seed([18C6ABB7085355A5:696BCB81DBED95DD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
	at org.apache.solr.cloud.CreateRoutedAliasTest.testCollectionNamesMustBeAbsent(CreateRoutedAliasTest.java:247)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.common.util.TestTimeSource.testEpochTime

Error Message:
SimTimeSource:50.0 time diff=30168800

Stack Trace:
java.lang.AssertionError: SimTimeSource:50.0 time diff=30168800
	at __randomizedtesting.SeedInfo.seed([27EE123CA4A3F1B3:1F826119307353F5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.common.util.TestTimeSource.doTestEpochTime(TestTimeSource.java:48)
	at org.apache.solr.common.util.TestTimeSource.testEpochTime(TestTimeSource.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 1795 lines...]
   [junit4] JVM J1: stdout was not empty, see: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\build\core\test\temp\junit4-J1-20180325_082124_4365253096933869104684.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] codec: CheapBastard, pf: BloomFilter, dvf: Memory
   [junit4] <<< JVM J1: EOF ----

[...truncated 11574 lines...]
   [junit4] Suite: org.apache.solr.cloud.CreateRoutedAliasTest
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.CreateRoutedAliasTest_18C6ABB7085355A5-001\init-core-data-001
   [junit4]   2> 1894939 WARN  (SUITE-CreateRoutedAliasTest-seed#[18C6ABB7085355A5]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=12 numCloses=12
   [junit4]   2> 1894940 INFO  (SUITE-CreateRoutedAliasTest-seed#[18C6ABB7085355A5]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
   [junit4]   2> 1894968 INFO  (SUITE-CreateRoutedAliasTest-seed#[18C6ABB7085355A5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1894968 INFO  (SUITE-CreateRoutedAliasTest-seed#[18C6ABB7085355A5]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1894988 INFO  (SUITE-CreateRoutedAliasTest-seed#[18C6ABB7085355A5]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.CreateRoutedAliasTest_18C6ABB7085355A5-001\tempDir-001
   [junit4]   2> 1894992 INFO  (SUITE-CreateRoutedAliasTest-seed#[18C6ABB7085355A5]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1894993 INFO  (Thread-2724) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1894993 INFO  (Thread-2724) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1895001 ERROR (Thread-2724) [    ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1895093 INFO  (SUITE-CreateRoutedAliasTest-seed#[18C6ABB7085355A5]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:62730
   [junit4]   2> 1895101 INFO  (zkConnectionManagerCallback-3433-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895111 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10004a5e83d0000, likely client has closed socket
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1d56c9b{/solr,null,AVAILABLE}
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T17:27:37-04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.e.j.s.session Scavenging every 600000ms
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@ac1b4d{HTTP/1.1,[http/1.1]}{127.0.0.1:62734}
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.e.j.s.Server Started @1903010ms
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=62734}
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@756874{/solr,null,AVAILABLE}
   [junit4]   2> 1895120 ERROR (jetty-launcher-3430-thread-1) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-25T09:11:46.163Z
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@442361{HTTP/1.1,[http/1.1]}{127.0.0.1:62737}
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.e.j.s.Server Started @1903012ms
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=62737}
   [junit4]   2> 1895120 ERROR (jetty-launcher-3430-thread-2) [    ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.4.0
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1895120 INFO  (jetty-launcher-3430-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2018-03-25T09:11:46.163Z
   [junit4]   2> 1895133 INFO  (zkConnectionManagerCallback-3435-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895133 INFO  (zkConnectionManagerCallback-3437-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895135 INFO  (jetty-launcher-3430-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1895135 INFO  (jetty-launcher-3430-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1895167 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10004a5e83d0001, likely client has closed socket
   [junit4]   2> 1895171 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10004a5e83d0002, likely client has closed socket
   [junit4]   2> 1895178 INFO  (jetty-launcher-3430-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62730/solr
   [junit4]   2> 1895178 INFO  (jetty-launcher-3430-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:62730/solr
   [junit4]   2> 1895181 INFO  (zkConnectionManagerCallback-3443-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895182 INFO  (zkConnectionManagerCallback-3445-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895183 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10004a5e83d0004, likely client has closed socket
   [junit4]   2> 1895186 INFO  (zkConnectionManagerCallback-3449-thread-1-processing-n:127.0.0.1:62737_solr) [n:127.0.0.1:62737_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895186 INFO  (zkConnectionManagerCallback-3447-thread-1-processing-n:127.0.0.1:62734_solr) [n:127.0.0.1:62734_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895476 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1895476 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1895477 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:62737_solr
   [junit4]   2> 1895478 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62734_solr
   [junit4]   2> 1895478 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.c.Overseer Overseer (id=72062704650354693-127.0.0.1:62737_solr-n_0000000000) starting
   [junit4]   2> 1895485 INFO  (zkCallback-3446-thread-1-processing-n:127.0.0.1:62734_solr) [n:127.0.0.1:62734_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1895485 INFO  (zkCallback-3448-thread-1-processing-n:127.0.0.1:62737_solr) [n:127.0.0.1:62737_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1895491 INFO  (zkConnectionManagerCallback-3455-thread-1-processing-n:127.0.0.1:62737_solr) [n:127.0.0.1:62737_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1895493 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1895494 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:62730/solr ready
   [junit4]   2> 1895498 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62737_solr
   [junit4]   2> 1895503 INFO  (zkCallback-3446-thread-1-processing-n:127.0.0.1:62734_solr) [n:127.0.0.1:62734_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1895503 INFO  (zkCallback-3448-thread-1-processing-n:127.0.0.1:62737_solr) [n:127.0.0.1:62737_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1895511 INFO  (zkCallback-3454-thread-1-processing-n:127.0.0.1:62737_solr) [n:127.0.0.1:62737_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1895997 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62734.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5f305
   [junit4]   2> 1896014 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62734.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5f305
   [junit4]   2> 1896015 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62734.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5f305
   [junit4]   2> 1896018 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.CreateRoutedAliasTest_18C6ABB7085355A5-001\tempDir-001\node2\.
   [junit4]   2> 1896027 INFO  (zkConnectionManagerCallback-3459-thread-1-processing-n:127.0.0.1:62734_solr) [n:127.0.0.1:62734_solr    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1896030 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1896031 INFO  (jetty-launcher-3430-thread-1) [n:127.0.0.1:62734_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:62730/solr ready
   [junit4]   2> 1896498 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62737.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5f305
   [junit4]   2> 1896524 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62737.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5f305
   [junit4]   2> 1896525 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62737.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5f305
   [junit4]   2> 1896527 INFO  (jetty-launcher-3430-thread-2) [n:127.0.0.1:62737_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.CreateRoutedAliasTest_18C6ABB7085355A5-001\tempDir-001\node1\.
   [junit4]   2> 1896573 INFO  (zkConnectionManagerCallback-3461-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1896574 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10004a5e83d0009, likely client has closed socket
   [junit4]   2> 1896599 INFO  (TEST-CreateRoutedAliasTest.testNegativeFutureFails-seed#[18C6ABB7085355A5]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testNegativeFutureFails
   [junit4]   2> 1896606 INFO  (zkConnectionManagerCallback-3466-thread-1) [    ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1896608 INFO  (TEST-CreateRoutedAliasTest.testNegativeFutureFails-seed#[18C6ABB7085355A5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1896610 INFO  (TEST-CreateRoutedAliasTest.testNegativeFutureFails-seed#[18C6ABB7085355A5]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:62730/solr ready
   [junit4]   2> 1896614 INFO  (TEST-CreateRoutedAliasTest.testNegativeFutureFails-seed#[18C6ABB7085355A5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1896620 INFO  (qtp26492995-12923) [n:127.0.0.1:62734_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :createalias with params router.interval=%2B30MINUTE&router.start=2018-01-15T00:00:00Z&create-collection.collection.configName=_default&router.field=evt_dt&name=testNegativeFutureFails&router.name=time&create-collection.numShards=1&action=CREATEALIAS&wt=json&router.maxFutureMs=-60000 and sendToOCPQueue=true
   [junit4]   2> 1896623 ERROR (OverseerThreadFactory-4640-thread-1-processing-n:127.0.0.1:62737_solr) [n:127.0.0.1:62737_solr    ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: testNegativeFutureFails operation: createalias failed:org.apache.solr.common.SolrException: router.maxFutureMs must be >= 0
   [junit4]   2> 	at org.apache.solr.cloud.api.collections.TimeRoutedAlias.<init>(TimeRoutedAlias.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.api.collections.CreateAliasCmd.callCreateRoutedAlias(CreateAliasCmd.java:116)
   [junit4]   2> 	at org.apache.solr.cloud.api.collections.CreateAliasCmd.call(CreateAliasCmd.java:65)
   [junit4]   2> 	at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:252)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:469)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> 1896633 INFO  (qtp26492995-12923) [n:127.0.0.1:62734_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={router.interval=%2B30MINUTE&router.start=2018-01-15T00:00:00Z&create-collection.collection.configName=_default&router.field=evt_dt&name=testNegativeFutureFails&router.name=time&create-collection.numShards=1&action=CREATEALIAS&wt=json&router.maxFutureMs=-60000} status=400 QTime=14
   [junit4]   2> 1896635 INFO  (TEST-CreateRoutedAliasTest.testNegativeFutureFails-seed#[18C6ABB7085355A5]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testNegativeFutureFails
   [junit4]   2> 1896668 INFO  (TEST-CreateRoutedAliasTest.testV1-seed#[18C6ABB7085355A5]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testV1
   [junit4]   2> 1896676 INFO  (TEST-CreateRoutedAliasTest.testV1-seed#[18C6ABB7085355A5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1896679 INFO  (qtp26492995-12928) [n:127.0.0.1:62734_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :createalias with params router.interval=%2B30MINUTE&router.start=2018-03-25T09:00:00Z&create-collection.collection.configName=_default&router.field=evt_dt&name=testV1&router.name=time&create-collection.numShards=1&action=CREATEALIAS&create-collection.router.field=foo_s&wt=xml&create-collection.replicationFactor=2 and sendToOCPQueue=true
   [junit4]   2> 1896685 INFO  (OverseerCollectionConfigSetProcessor-72062704650354693-127.0.0.1:62737_solr-n_0000000000) [n:127.0.0.1:62737_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1896685 INFO  (OverseerThreadFactory-4640-thread-2-processing-n:127.0.0.1:62737_solr) [n:127.0.0.1:62737_solr    ] o.a.s.c.a.c.CreateCollectionCmd Create collection testV1_2018-03-25_09
   [junit4]   2> 1896806 INFO  (OverseerStateUpdate-72062704650354693-127.0.0.1:62737_solr-n_0000000000) [n:127.0.0.1:62737_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testV1_2018-03-25_09",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testV1_2018-03-25_09_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:62734/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1896813 INFO  (OverseerStateUpdate-72062704650354693-127.0.0.1:62737_solr-n_0000000000) [n:127.0.0.1:62737_solr    ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"testV1_2018-03-25_09",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"testV1_2018-03-25_09_shard1_replica_n3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"http://127.0.0.1:62737/solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 1897020 INFO  (qtp7030043-12932) [n:127.0.0.1:62737_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=_default&newCollection=true&collection=testV1_2018-03-25_09&version=2&replicaType=NRT&property.routedAliasName=testV1&coreNodeName=core_node4&name=testV1_2018-03-25_09_shard1_replica_n3&action=CREATE&numSha

[...truncated too long message...]

ng 154 source files to C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-solrj\classes\test
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

compile-test:

BUILD SUCCESSFUL
Total time: 8 seconds
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestTimeSource" -Dtests.showOutput=onerror "-Dargs=-client -XX:+UseSerialGC" -Dtests.seed=27EE123CA4A3F1B3 -Dtests.slow=true -Dtests.locale=hi-IN -Dtests.timezone=Europe/Monaco -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
Buildfile: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\solrj\build.xml

-clover.disable:

ivy-configure:
[ivy:configure] :: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\top-level-ivy-settings.xml

install-junit4-taskdef:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

resolve-groovy:
[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] 	confs: [default]
[ivy:cachepath] 	found org.codehaus.groovy#groovy-all;2.4.13 in public
[ivy:cachepath] :: resolution report :: resolve 16ms :: artifacts dl 1ms
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
	---------------------------------------------------------------------

-init-totals:

-test:
    [mkdir] Created dir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-solrj\test
[junit4:pickseed] Seed property 'tests.seed' already defined: 27EE123CA4A3F1B3
    [mkdir] Created dir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-solrj\test\temp
   [junit4] <JUnit4> says ¡Hola! Master seed: 27EE123CA4A3F1B3
   [junit4] Your default console's encoding may not display certain unicode glyphs: windows-1252
   [junit4] Executing 5 suites with 2 JVMs.
   [junit4] 
   [junit4] Started J0 PID(20272@serv1-vm1).
   [junit4] Started J1 PID(26156@serv1-vm1).
   [junit4] Suite: org.apache.solr.common.util.TestTimeSource
   [junit4]   2> SLF4J: Class path contains multiple SLF4J bindings.
   [junit4]   2> SLF4J: Found binding in [jar:file:/C:/Users/jenkins/workspace/Lucene-Solr-7.x-Windows/solr/solrj/test-lib/slf4j-log4j12-1.7.24.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   [junit4]   2> SLF4J: Found binding in [jar:file:/C:/Users/jenkins/workspace/Lucene-Solr-7.x-Windows/solr/core/lib/slf4j-log4j12-1.7.24.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   [junit4]   2> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
   [junit4]   2> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
   [junit4]   2> 0    INFO  (SUITE-TestTimeSource-seed#[27EE123CA4A3F1B3]-worker) [    ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-solrj\test\J0\temp\solr.common.util.TestTimeSource_27EE123CA4A3F1B3-001\init-core-data-001
   [junit4]   2> 272  INFO  (SUITE-TestTimeSource-seed#[27EE123CA4A3F1B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 382  INFO  (SUITE-TestTimeSource-seed#[27EE123CA4A3F1B3]-worker) [    ] o.e.j.u.log Logging initialized @????ms to org.eclipse.jetty.util.log.Slf4jLog
   [junit4]   2> 391  INFO  (SUITE-TestTimeSource-seed#[27EE123CA4A3F1B3]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 425  INFO  (TEST-TestTimeSource.testEpochTime-seed#[27EE123CA4A3F1B3]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEpochTime
   [junit4]   2> 10454 INFO  (TEST-TestTimeSource.testEpochTime-seed#[27EE123CA4A3F1B3]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEpochTime
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestTimeSource -Dtests.method=testEpochTime -Dtests.seed=27EE123CA4A3F1B3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=hi-IN -Dtests.timezone=Europe/Monaco -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 10.1s J0 | TestTimeSource.testEpochTime <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: SimTimeSource:50.0 time diff=24181200
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([27EE123CA4A3F1B3:1F826119307353F5]:0)
   [junit4]    > 	at org.apache.solr.common.util.TestTimeSource.doTestEpochTime(TestTimeSource.java:48)
   [junit4]    > 	at org.apache.solr.common.util.TestTimeSource.testEpochTime(TestTimeSource.java:32)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-solrj\test\J0\temp\solr.common.util.TestTimeSource_27EE123CA4A3F1B3-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=hi-IN, timezone=Europe/Monaco
   [junit4]   2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_144 (32-bit)/cpus=3,threads=1,free=2790256,total=16252928
   [junit4]   2> NOTE: All tests run in this JVM: [TestTimeSource]
   [junit4] Completed [1/5 (1!)] on J0 in 16.77s, 1 test, 1 failure <<< FAILURES!
   [junit4] 
   [junit4] Suite: org.apache.solr.common.util.TestTimeSource
   [junit4] OK      10.2s J1 | TestTimeSource.testEpochTime
   [junit4] Completed [2/5 (1!)] on J1 in 16.98s, 1 test
   [junit4] 
   [junit4] Duplicate suite name used with XML reports: org.apache.solr.common.util.TestTimeSource. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message.
   [junit4] Suite: org.apache.solr.common.util.TestTimeSource
   [junit4] OK      10.1s J0 | TestTimeSource.testEpochTime
   [junit4] Completed [3/5 (1!)] on J0 in 10.16s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.common.util.TestTimeSource
   [junit4] OK      10.1s J1 | TestTimeSource.testEpochTime
   [junit4] Completed [4/5 (1!)] on J1 in 10.15s, 1 test
   [junit4] 
   [junit4] Suite: org.apache.solr.common.util.TestTimeSource
   [junit4] OK      10.1s J0 | TestTimeSource.testEpochTime
   [junit4] Completed [5/5 (1!)] on J0 in 10.14s, 1 test
   [junit4] 
   [junit4] 
   [junit4] Tests with failures [seed: 27EE123CA4A3F1B3]:
   [junit4]   - org.apache.solr.common.util.TestTimeSource.testEpochTime
   [junit4] 
   [junit4] 
   [junit4] JVM J0:     0.62 ..    38.59 =    37.97s
   [junit4] JVM J1:     0.62 ..    28.71 =    28.09s
   [junit4] Execution time total: 38 seconds
   [junit4] Tests summary: 5 suites, 5 tests, 1 failure

BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1561: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\lucene\common-build.xml:1085: There were test failures: 5 suites, 5 tests, 1 failure [seed: 27EE123CA4A3F1B3]

Total time: 39 seconds

[repro] Setting last failure code to 1

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.CreateRoutedAliasTest
[repro]   1/5 failed: org.apache.solr.common.util.TestTimeSource
[repro]   2/5 failed: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[repro] Exiting with code 1

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build move /y lucene\build lucene\build.repro 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build move /y solr\build solr\build.repro 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST lucene\build.orig move /y lucene\build.orig lucene\build 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>IF EXIST solr\build.orig move /y solr\build.orig solr\build 
        1 dir(s) moved.

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows>exit 0 
Archiving artifacts
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=67572cf6d3127df3e749019bfcdcef97c738c6b1, workspace=C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows
[WARNINGS] Computing warning deltas based on reference build #517
Recording test results
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2
Setting ANT_1_8_2_HOME=C:\Users\jenkins\tools\hudson.tasks.Ant_AntInstallation\ANT_1.8.2