You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2016/07/31 02:32:25 UTC

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1084 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1084/

No tests ran.

Build Log:
Started by timer

[EnvInject] - Loading node environment variables.

Building remotely on lucene in workspace /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master

 > git rev-parse --is-inside-work-tree # timeout=10

Fetching changes from the remote Git repository

 > git config remote.origin.url https://git-wip-us.apache.org/repos/asf/lucene-solr.git # timeout=10

Cleaning workspace

 > git rev-parse --verify HEAD # timeout=10

Resetting working tree

 > git reset --hard # timeout=10

 > git clean -fdx # timeout=10

Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/lucene-solr.git

 > git --version # timeout=10

 > git -c core.askpass=true fetch --tags --progress https://git-wip-us.apache.org/repos/asf/lucene-solr.git +refs/heads/*:refs/remotes/origin/*

 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10

 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10

Checking out Revision 53a34b312e78ce6f56c0bb41304ac834b28b9534 (refs/remotes/origin/master)

 > git config core.sparsecheckout # timeout=10

 > git checkout -f 53a34b312e78ce6f56c0bb41304ac834b28b9534

 > git rev-list 9554719f12080e233dbfe634f4a7066e138033ed # timeout=10

No emails were triggered.

[Lucene-Solr-NightlyTests-master] $ /home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/bin/ant -Dtests.multiplier=2 -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt jenkins-nightly

Buildfile: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml


jenkins-nightly:


-print-java-info:

[java-info] java version "1.8.0_74"

[java-info] Java(TM) SE Runtime Environment (1.8.0_74-b02, Oracle Corporation)

[java-info] Java HotSpot(TM) 64-Bit Server VM (25.74-b02, Oracle Corporation)

[java-info] Test args: []


clean:


clean:


clean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve-groovy:

[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working

[ivy:cachepath] 	confs: [default]

[ivy:cachepath] 	found org.codehaus.groovy#groovy-all;2.4.6 in public

[ivy:cachepath] :: resolution report :: resolve 89ms :: artifacts dl 2ms

	---------------------------------------------------------------------

	|                  |            modules            ||   artifacts   |

	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|

	---------------------------------------------------------------------

	|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |

	---------------------------------------------------------------------


-test-with-heapdumps-enabled:

     [echo] Java HotSpot(TM) 64-Bit Server VM: Enabling heap dumps on OutOfMemoryError to dir '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps'.

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve-groovy:


test:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


-clover.load:


resolve-groovy:


-init-totals:


test-core:


-clover.disable:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


-clover.load:


-clover.classpath:


-clover.setup:


clover:


-check-git-state:


-git-cleanroot:


-copy-git-state:

     [copy] Copying 1 file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build


git-autoclean:


resolve:


init:


compile-core:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java

    [javac] Compiling 779 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java

    [javac] Note: Some input files use or override a deprecated API.

    [javac] Note: Recompile with -Xlint:deprecation for details.

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/document/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/bkd/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/compressing/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene60/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/index/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/fst/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/tokenattributes/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/search/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene62/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/packed/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/mutable/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/perfield/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/standard/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/store/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/blocktree/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/search/spans/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene54/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/search/similarities/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/automaton/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene53/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene50/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/geo/package-info.class

     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java


compile-test-framework:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


init:


compile-lucene-core:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


init:


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


compile-core:


compile-codecs:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


common.init:


compile-lucene-core:


init:


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


compile-core:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java

    [javac] Compiling 66 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/bloom/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/memory/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/simpletext/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/blockterms/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/blocktreeords/package-info.class

     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


common.compile-core:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java

    [javac] Compiling 185 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java

    [javac] Note: Some input files use or override a deprecated API.

    [javac] Note: Recompile with -Xlint:deprecation for details.

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/asserting/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/ramonly/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/mockfile/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/cheapbastard/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/mockrandom/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/cranky/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/compressing/dummy/package-info.class

     [copy] Copying 4 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java


compile-core:


compile-test:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/test

    [javac] Compiling 444 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/test

    [javac] Note: Some input files use or override a deprecated API.

    [javac] Note: Recompile with -Xlint:deprecation for details.

     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/test


install-junit4-taskdef:


validate:


resolve-groovy:


-init-totals:


-test:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/test

[loadresource] Do not set property tests.explicitclass as its length is 0.

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/test/temp

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/.caches/test-stats/core

   [junit4] <JUnit4> says olá! Master seed: C1DD342640B5FD06

   [junit4] Executing 437 suites with 3 JVMs.

   [junit4] 

   [junit4] Started J1 PID(14712@localhost).

   [junit4] Started J0 PID(14711@localhost).

   [junit4] Started J2 PID(14713@localhost).

   [junit4] Suite: org.apache.lucene.search.TestPositionIncrement

   [junit4] Completed [1/437] on J1 in 1.00s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSortedSetDocValues

   [junit4] Completed [2/437] on J1 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanNearQuery

   [junit4] Completed [3/437] on J1 in 0.10s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexReaderClose

   [junit4] Completed [4/437] on J0 in 2.04s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestAllFilesHaveCodecHeader

   [junit4] Completed [5/437] on J1 in 1.12s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestPositiveScoresOnlyCollector

   [junit4] Completed [6/437] on J1 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSynonymQuery

   [junit4] Completed [7/437] on J1 in 0.06s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BSortedDocValuesOrds

   [junit4] IGNOR/A 0.00s J1 | Test2BSortedDocValuesOrds.test2BOrds

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=Takes ~ 6 hours if the heap is 5gb))

   [junit4] Completed [8/437] on J1 in 0.01s, 1 test, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.similarities.TestBM25Similarity

   [junit4] Completed [9/437] on J1 in 0.04s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterConfig

   [junit4] Completed [10/437] on J1 in 0.11s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestNumericRangeQuery64

   [junit4] Completed [11/437] on J0 in 27.50s, 29 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestCustomNorms

   [junit4] Completed [12/437] on J0 in 5.32s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiset

   [junit4] Completed [13/437] on J0 in 0.03s, 2 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:29:52, stalled for 69.9s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanSearchEquivalence

   [junit4] Completed [14/437] on J0 in 39.85s, 33 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestUsageTrackingFilterCachingPolicy

   [junit4] Completed [15/437] on J0 in 0.02s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanContainQuery

   [junit4] Completed [16/437] on J0 in 0.22s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestExitableDirectoryReader

   [junit4] Completed [17/437] on J0 in 0.45s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:30:04, stalled for 71.7s at: TestPackedInts.testPackedLongValues

   [junit4] Suite: org.apache.lucene.util.packed.TestPackedInts

   [junit4] Completed [18/437] on J1 in 106.49s, 28 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestVersion

   [junit4] Completed [19/437] on J1 in 0.32s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBooleanQueryVisitSubscorers

   [junit4] Completed [20/437] on J1 in 0.08s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestThreadedForceMerge

   [junit4] Completed [21/437] on J1 in 2.94s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestTrackingDirectoryWrapper

   [junit4] IGNOR/A 0.00s J1 | TestTrackingDirectoryWrapper.testFsyncDoesntCreateNewFiles

   [junit4]    > Assumption #1: test only works for FSDirectory subclasses

   [junit4] IGNOR/A 0.00s J1 | TestTrackingDirectoryWrapper.testPendingDeletions

   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory

   [junit4] Completed [22/437] on J1 in 1.87s, 49 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.document.TestDocument

   [junit4] Completed [23/437] on J1 in 0.21s, 12 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestSetOnce

   [junit4] Completed [24/437] on J1 in 0.06s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50StoredFieldsFormatHighCompression

   [junit4] Completed [25/437] on J0 in 45.16s, 19 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BPostingsBytes

   [junit4] IGNOR/A 0.01s J0 | Test2BPostingsBytes.test

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes ~20GB-30GB of space and 10 minutes))

   [junit4] Completed [26/437] on J0 in 0.02s, 1 test, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMinShouldMatch2

   [junit4] Completed [27/437] on J0 in 5.19s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.document.TestDateTools

   [junit4] Completed [28/437] on J0 in 0.05s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestSimpleFSDirectory

   [junit4] Completed [29/437] on J0 in 1.35s, 44 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:30:52, stalled for  130s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] Suite: org.apache.lucene.index.TestAllFilesDetectTruncation

   [junit4] Completed [30/437] on J0 in 8.84s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestCrashCausesCorruptIndex

   [junit4] Completed [31/437] on J0 in 0.18s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNRTReaderWithThreads

   [junit4] Completed [32/437] on J0 in 1.45s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBooleanScorer

   [junit4] Completed [33/437] on J0 in 5.01s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestStringHelper

   [junit4] Completed [34/437] on J0 in 0.04s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.geo.TestPolygon2D

   [junit4] Completed [35/437] on J1 in 28.77s, 10 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNoMergePolicy

   [junit4] Completed [36/437] on J1 in 0.13s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMergePolicy

   [junit4] Completed [37/437] on J0 in 4.28s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene60.TestLucene60PointsFormat

   [junit4] IGNOR/A 0.00s J1 | TestLucene60PointsFormat.testMergeStability

   [junit4]    > Assumption #1: TODO: mess with the parameters and test gets angry!

   [junit4] Completed [38/437] on J1 in 28.45s, 17 tests, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanBoostQuery

   [junit4] Completed [39/437] on J1 in 0.03s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterLockRelease

   [junit4] Completed [40/437] on J1 in 0.06s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:31:52, stalled for  190s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:32:28, stalled for 63.8s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:32:52, stalled for 63.7s at: TestIndexSorting.testRandom3

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:32:52, stalled for  250s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:33:28, stalled for  124s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:33:52, stalled for  124s at: TestIndexSorting.testRandom3

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:33:52, stalled for  310s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:34:28, stalled for  184s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:34:52, stalled for  184s at: TestIndexSorting.testRandom3

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:34:52, stalled for  370s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] Suite: org.apache.lucene.index.TestIndexSorting

   [junit4] Completed [41/437] on J1 in 231.47s, 27 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDoc

   [junit4] Completed [42/437] on J1 in 0.17s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSameScoresWithThreads

   [junit4] Completed [43/437] on J1 in 0.78s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:35:28, stalled for  244s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.util.TestFixedBitSet

   [junit4] Completed [44/437] on J1 in 19.47s, 19 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:35:52, stalled for  430s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:36:28, stalled for  304s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:36:52, stalled for  490s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:37:04, stalled for 71.0s at: TestDuelingCodecsAtNight.testBigEquals

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:37:28, stalled for  364s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:37:52, stalled for  550s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:38:04, stalled for  131s at: TestDuelingCodecsAtNight.testBigEquals

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:38:28, stalled for  424s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:38:52, stalled for  610s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:39:04, stalled for  191s at: TestDuelingCodecsAtNight.testBigEquals

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:39:28, stalled for  484s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:39:52, stalled for  670s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:40:04, stalled for  251s at: TestDuelingCodecsAtNight.testBigEquals

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:40:28, stalled for  544s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:40:52, stalled for  730s at: TestDocumentsWriterStallControl.testAccquireReleaseRace

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:41:04, stalled for  311s at: TestDuelingCodecsAtNight.testBigEquals

   [junit4] Suite: org.apache.lucene.index.TestDocumentsWriterStallControl

   [junit4] Completed [45/437] on J2 in 752.47s, 3 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:41:28, stalled for  604s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:42:04, stalled for  371s at: TestDuelingCodecsAtNight.testBigEquals

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:42:28, stalled for  664s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.search.TestEarlyTerminatingSortingCollector

   [junit4] Completed [46/437] on J2 in 79.10s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestBagOfPostings

   [junit4] Completed [47/437] on J2 in 6.47s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestCustomSearcherSort

   [junit4] Completed [48/437] on J2 in 11.81s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.similarities.TestClassicSimilarity

   [junit4] Completed [49/437] on J2 in 0.14s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSloppyPhraseQuery

   [junit4] Completed [50/437] on J2 in 3.03s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestTermQuery

   [junit4] Completed [51/437] on J2 in 0.05s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocInverterPerFieldErrorInfo

   [junit4] Completed [52/437] on J2 in 0.30s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDuelingCodecsAtNight

   [junit4] Completed [53/437] on J1 in 431.37s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestStopFilter

   [junit4] Completed [54/437] on J1 in 0.05s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50StoredFieldsFormat

   [junit4] Completed [55/437] on J2 in 29.16s, 17 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BPoints

   [junit4] IGNOR/A 0.00s J2 | Test2BPoints.test1D

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes at least 4 hours and consumes many GB of temp disk space))

   [junit4] IGNOR/A 0.00s J2 | Test2BPoints.test2D

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes at least 4 hours and consumes many GB of temp disk space))

   [junit4] Completed [56/437] on J2 in 0.02s, 2 tests, 2 skipped

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:43:28, stalled for  724s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.search.TestBooleanOr

   [junit4] Completed [57/437] on J2 in 10.75s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene62.TestLucene62SegmentInfoFormat

   [junit4] Completed [58/437] on J2 in 0.47s, 16 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestByteArrayDataInput

   [junit4] Completed [59/437] on J2 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestAllFilesCheckIndexHeader

   [junit4] Completed [60/437] on J2 in 4.32s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:44:28, stalled for  784s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:44:40, stalled for 60.9s at: TestBagOfPositions.test

   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat

   [junit4] Completed [61/437] on J1 in 104.83s, 96 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestStressAdvance

   [junit4] Completed [62/437] on J1 in 10.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBooleanQuery

   [junit4] Completed [63/437] on J1 in 0.73s, 22 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSegmentReader

   [junit4] Completed [64/437] on J1 in 0.65s, 7 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:45:28, stalled for  844s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:45:40, stalled for  121s at: TestBagOfPositions.test

   [junit4] Suite: org.apache.lucene.index.TestBagOfPositions

   [junit4] Completed [65/437] on J2 in 141.23s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:46:04, stalled for 71.6s at: TestGrowableByteArrayDataOutput.testWriteLargeStrings

   [junit4] Suite: org.apache.lucene.codecs.compressing.TestGrowableByteArrayDataOutput

   [junit4] Completed [66/437] on J1 in 79.63s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.automaton.TestOperations

   [junit4] Completed [67/437] on J1 in 0.61s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestSparseFixedBitDocIdSet

   [junit4] Completed [68/437] on J1 in 1.38s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestRegexpQuery

   [junit4] Completed [69/437] on J1 in 0.14s, 7 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:46:28, stalled for  904s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat

   [junit4] IGNOR/A 0.01s J1 | TestPerFieldPostingsFormat.testMergeStability

   [junit4]    > Assumption #1: The MockRandom PF randomizes content on the fly, so we can't check it

   [junit4] IGNOR/A 0.02s J1 | TestPerFieldPostingsFormat.testPostingsEnumReuse

   [junit4]    > Assumption #1: The MockRandom PF randomizes content on the fly, so we can't check it

   [junit4] Completed [70/437] on J1 in 18.80s, 24 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestMergedIterator

   [junit4] Completed [71/437] on J1 in 0.36s, 25 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestAttributeSource

   [junit4] Completed [72/437] on J1 in 0.02s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiCollector

   [junit4] Completed [73/437] on J1 in 1.92s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestRoaringDocIdSet

   [junit4] Completed [74/437] on J1 in 0.88s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNewestSegment

   [junit4] Completed [75/437] on J1 in 0.06s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestFilterDirectory

   [junit4] IGNOR/A 0.00s J1 | TestFilterDirectory.testPendingDeletions

   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory

   [junit4] IGNOR/A 0.00s J1 | TestFilterDirectory.testFsyncDoesntCreateNewFiles

   [junit4]    > Assumption #1: test only works for FSDirectory subclasses

   [junit4] Completed [76/437] on J1 in 1.09s, 46 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestParallelTermEnum

   [junit4] Completed [77/437] on J1 in 0.07s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterCommit

   [junit4] Completed [78/437] on J1 in 2.21s, 14 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestBasics

   [junit4] Completed [79/437] on J1 in 3.19s, 25 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestInfoStream

   [junit4] Completed [80/437] on J1 in 0.06s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpans

   [junit4] Completed [81/437] on J1 in 0.82s, 26 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestByteSlices

   [junit4] Completed [82/437] on J1 in 5.15s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSloppyPhraseQuery2

   [junit4] Completed [83/437] on J1 in 10.99s, 9 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:47:04, stalled for 63.2s at: TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:47:28, stalled for  964s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.index.TestNumericDocValuesUpdates

   [junit4] Completed [84/437] on J1 in 28.81s, 31 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestCharArraySet

   [junit4] Completed [85/437] on J1 in 0.08s, 15 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSearchWithThreads

   [junit4] Completed [86/437] on J1 in 18.66s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.compressing.TestFastCompressionMode

   [junit4] Completed [87/437] on J1 in 1.26s, 11 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:48:04, stalled for  123s at: TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50TermVectorsFormat

   [junit4] Completed [88/437] on J1 in 29.34s, 17 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.similarities.TestSimilarityBase

   [junit4] Completed [89/437] on J1 in 1.29s, 32 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanExplanations

   [junit4] Completed [90/437] on J1 in 0.36s, 30 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMultiDocValues

   [junit4] Completed [91/437] on J1 in 5.91s, 8 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:48:28, stalled for 1024s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.store.TestNativeFSLockFactory

   [junit4] Completed [92/437] on J1 in 8.42s, 10 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.bkd.TestBKD

   [junit4] Completed [93/437] on J1 in 17.73s, 16 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestByteBlockPool

   [junit4] Completed [94/437] on J1 in 0.09s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestFieldReuse

   [junit4] Completed [95/437] on J1 in 0.03s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestFrequencyTrackingRingBuffer

   [junit4] Completed [96/437] on J1 in 0.48s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterUnicode

   [junit4] Completed [97/437] on J1 in 4.04s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestFileSwitchDirectory

   [junit4] IGNOR/A 0.01s J1 | TestFileSwitchDirectory.testPendingDeletions

   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory

   [junit4] IGNOR/A 0.01s J1 | TestFileSwitchDirectory.testFsyncDoesntCreateNewFiles

   [junit4]    > Assumption #1: test only works for FSDirectory subclasses

   [junit4] Completed [98/437] on J1 in 2.39s, 45 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestBlockPostingsFormat3

   [junit4] Completed [99/437] on J1 in 4.30s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:49:04, stalled for  183s at: TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:49:28, stalled for 1084s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:50:04, stalled for  243s at: TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterThreadsToSegments

   [junit4] Completed [100/437] on J2 in 250.13s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestStressIndexing2

   [junit4] Completed [101/437] on J2 in 2.74s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestTimSorter

   [junit4] Completed [102/437] on J2 in 0.42s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestFlex

   [junit4] Completed [103/437] on J2 in 0.24s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestForceMergeForever

   [junit4] Completed [104/437] on J2 in 0.67s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BBinaryDocValues

   [junit4] IGNOR/A 0.00s J2 | Test2BBinaryDocValues.testFixedBinary

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes ~ 6 hours if the heap is 5gb))

   [junit4] IGNOR/A 0.00s J2 | Test2BBinaryDocValues.testVariableBinary

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes ~ 6 hours if the heap is 5gb))

   [junit4] Completed [105/437] on J2 in 0.01s, 2 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiThreadTermVectors

   [junit4] Completed [106/437] on J2 in 1.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestOmitPositions

   [junit4] Completed [107/437] on J2 in 0.46s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestFilterLeafReader

   [junit4] Completed [109/437] on J2 in 0.08s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestLogMergePolicy

   [junit4] Completed [110/437] on J2 in 0.16s, 2 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:50:28, stalled for 1144s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.index.TestConcurrentMergeScheduler

   [junit4] Completed [111/437] on J2 in 16.95s, 16 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestElevationComparator

   [junit4] Completed [112/437] on J2 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestAutomatonQueryUnicode

   [junit4] Completed [113/437] on J2 in 0.09s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSegmentTermEnum

   [junit4] Completed [114/437] on J2 in 0.10s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestSingleInstanceLockFactory

   [junit4] Completed [115/437] on J2 in 2.46s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSimilarity

   [junit4] Completed [116/437] on J2 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestParallelReaderEmptyIndex

   [junit4] Completed [117/437] on J2 in 0.10s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMatchNoDocsQuery

   [junit4] Completed [118/437] on J2 in 0.04s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestConsistentFieldNumbers

   [junit4] Completed [119/437] on J2 in 3.20s, 4 tests

   [junit4] 

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:50:40, stalled for 66.4s at: TestFSTs.testBigSet

   [junit4] Suite: org.apache.lucene.util.fst.TestFSTs

   [junit4] Completed [120/437] on J1 in 106.86s, 20 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestPrefixRandom

   [junit4] Completed [121/437] on J1 in 1.50s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs

   [junit4] Completed [122/437] on J2 in 21.94s, 70 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestFilterDirectoryReader

   [junit4] Completed [123/437] on J2 in 0.07s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestLongBitSet

   [junit4] Completed [124/437] on J1 in 13.95s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BNumericDocValues

   [junit4] IGNOR/A 0.00s J1 | Test2BNumericDocValues.testNumerics

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes ~ 2 hours if the heap is 5gb))

   [junit4] Completed [125/437] on J1 in 0.00s, 1 test, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocumentsWriterDeleteQueue

   [junit4] Completed [126/437] on J1 in 0.14s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTwoPhaseCommitTool

   [junit4] Completed [127/437] on J1 in 0.01s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterFromReader

   [junit4] Completed [128/437] on J1 in 1.86s, 12 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMaxDocs

   [junit4] IGNOR/A 0.00s J2 | TestIndexWriterMaxDocs.testExactlyAtTrueLimit

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes over two hours))

   [junit4] Completed [129/437] on J2 in 11.38s, 21 tests, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestNRTCachingDirectory

   [junit4] IGNOR/A 0.00s J2 | TestNRTCachingDirectory.testPendingDeletions

   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory

   [junit4] IGNOR/A 0.00s J2 | TestNRTCachingDirectory.testFsyncDoesntCreateNewFiles

   [junit4]    > Assumption #1: test only works for FSDirectory subclasses

   [junit4] Completed [130/437] on J2 in 1.00s, 46 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSimilarityProvider

   [junit4] Completed [131/437] on J2 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestNotDocIdSet

   [junit4] Completed [132/437] on J2 in 1.77s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexFileDeleter

   [junit4] Completed [133/437] on J2 in 0.96s, 10 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexableField

   [junit4] Completed [134/437] on J2 in 0.61s, 2 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:51:29, stalled for 1204s at: TestIndexWriterOnVMError.testCheckpoint

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnVMError

   [junit4] Completed [135/437] on J0 in 1225.77s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BPositions

   [junit4] IGNOR/A 0.00s J0 | Test2BPositions.test

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=uses lots of space and takes a few minutes))

   [junit4] Completed [136/437] on J0 in 0.00s, 1 test, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSearchAfter

   [junit4] Completed [137/437] on J0 in 1.77s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSizeBoundedForceMerge

   [junit4] Completed [138/437] on J0 in 0.16s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.bkd.Test2BBKDPoints

   [junit4] IGNOR/A 0.00s J0 | Test2BBKDPoints.test1D

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes at least 4 hours and consumes many GB of temp disk space))

   [junit4] IGNOR/A 0.00s J0 | Test2BBKDPoints.test2D

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=takes at least 4 hours and consumes many GB of temp disk space))

   [junit4] Completed [139/437] on J0 in 0.02s, 2 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterExceptions2

   [junit4] Completed [140/437] on J0 in 11.79s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMixedDocValuesUpdates

   [junit4] Completed [141/437] on J1 in 40.16s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestOmitNorms

   [junit4] Completed [142/437] on J0 in 3.79s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocValues

   [junit4] Completed [143/437] on J0 in 0.15s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriter

   [junit4] Completed [144/437] on J1 in 8.87s, 71 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestPhraseQuery

   [junit4] Completed [145/437] on J0 in 5.83s, 21 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestIndexSearcher

   [junit4] Completed [146/437] on J0 in 1.09s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestAutomatonQuery

   [junit4] Completed [147/437] on J0 in 2.41s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestBytesRefHash

   [junit4] Completed [148/437] on J0 in 3.86s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestTermRangeQuery

   [junit4] Completed [149/437] on J0 in 0.25s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSwappedIndexFiles

   [junit4] Completed [150/437] on J0 in 0.17s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestPrefixQuery

   [junit4] Completed [151/437] on J0 in 10.86s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocValuesIndexing

   [junit4] Completed [152/437] on J0 in 0.51s, 34 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestTotalHitCountCollector

   [junit4] Completed [153/437] on J0 in 0.03s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:52:29, stalled for 71.4s at: Test4GBStoredFields.test

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterDelete

   [junit4] Completed [154/437] on J0 in 14.07s, 27 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestManyFields

   [junit4] Completed [155/437] on J0 in 7.48s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestArrayUtil

   [junit4] Completed [156/437] on J0 in 5.63s, 14 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestTermScorer

   [junit4] Completed [157/437] on J0 in 0.07s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestNumericUtils

   [junit4] Completed [158/437] on J0 in 4.52s, 24 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterWithThreads

   [junit4] IGNOR/A 0.03s J0 | TestIndexWriterWithThreads.testOpenTwoIndexWritersOnDifferentThreads

   [junit4]    > Assumption #1: aborting test: timeout obtaining lock

   [junit4] Completed [159/437] on J0 in 9.18s, 12 tests, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNeverDelete

   [junit4] Completed [160/437] on J0 in 5.74s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BTerms

   [junit4] IGNOR/A 0.00s J0 | Test2BTerms.test2BTerms

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=very slow, use 5g minimum heap))

   [junit4] Completed [161/437] on J0 in 0.01s, 1 test, 1 skipped

   [junit4] 

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:53:05, stalled for 62.1s at: TestIndexWriterReader.testDuringAddDelete

   [junit4] Suite: org.apache.lucene.util.automaton.TestUTF32ToUTF8

   [junit4] Completed [162/437] on J0 in 6.72s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocsAndPositions

   [junit4] Completed [163/437] on J0 in 0.62s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.geo.TestPolygon

   [junit4] Completed [164/437] on J0 in 0.04s, 15 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestRollback

   [junit4] Completed [165/437] on J0 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestComplexExplanationsOfNonMatches

   [junit4] Completed [166/437] on J0 in 0.06s, 21 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestSleepingLockWrapper

   [junit4] Completed [167/437] on J0 in 7.34s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.tokenattributes.TestCharTermAttributeImpl

   [junit4] Completed [168/437] on J0 in 0.49s, 12 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestCloseableThreadLocal

   [junit4] Completed [169/437] on J0 in 0.02s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSortedSetSortField

   [junit4] Completed [170/437] on J0 in 0.03s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestStressIndexing

   [junit4] Completed [171/437] on J0 in 4.47s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestFixedLengthBytesRefArray

   [junit4] Completed [172/437] on J0 in 0.27s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.document.TestFieldType

   [junit4] Completed [173/437] on J0 in 0.01s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestFieldsReader

   [junit4] Completed [174/437] on J0 in 0.32s, 2 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:53:29, stalled for  132s at: Test4GBStoredFields.test

   [junit4] Suite: org.apache.lucene.util.packed.TestDirectPacked

   [junit4] Completed [175/437] on J0 in 11.45s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTragicIndexWriterDeadlock

   [junit4] Completed [176/437] on J0 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestSmallFloat

   [junit4] Completed [177/437] on J0 in 0.14s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTermVectorsReader

   [junit4] Completed [178/437] on J0 in 0.08s, 13 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestBlockPostingsFormat2

   [junit4] Completed [179/437] on J0 in 6.87s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIntBlockPool

   [junit4] Completed [180/437] on J0 in 1.00s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.Test2BSortedDocValuesFixedSorted

   [junit4] IGNOR/A 0.00s J0 | Test2BSortedDocValuesFixedSorted.testFixedSorted

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=Takes ~ 6 hours if the heap is 5gb))

   [junit4] Completed [181/437] on J0 in 0.00s, 1 test, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSortRescorer

   [junit4] Completed [182/437] on J0 in 2.93s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestLiveFieldValues

   [junit4] Completed [183/437] on J0 in 1.25s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestParallelCompositeReader

   [junit4] Completed [184/437] on J0 in 0.60s, 15 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestPointValues

   [junit4] Completed [185/437] on J0 in 4.79s, 32 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestFieldCacheRewriteMethod

   [junit4] Completed [186/437] on J0 in 4.28s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestCachingCollector

   [junit4] Completed [187/437] on J0 in 0.17s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestCachingTokenFilter

   [junit4] Completed [188/437] on J0 in 0.02s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestComplexExplanations

   [junit4] Completed [189/437] on J0 in 0.44s, 21 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestLSBRadixSorter

   [junit4] Completed [190/437] on J0 in 0.09s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging

   [junit4] Completed [191/437] on J0 in 11.06s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.fst.TestBytesStore

   [junit4] Completed [192/437] on J0 in 9.84s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader

   [junit4] Completed [193/437] on J1 in 142.34s, 23 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestNumericRangeQuery32

   [junit4] Completed [194/437] on J0 in 4.19s, 23 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestAddIndexes

   [junit4] Completed [195/437] on J1 in 5.88s, 25 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:54:29, stalled for  192s at: Test4GBStoredFields.test

   [junit4] Suite: org.apache.lucene.search.TestShardSearching

   [junit4] Completed [196/437] on J0 in 16.70s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestMultiMMap

   [junit4] Completed [197/437] on J0 in 7.60s, 56 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:55:29, stalled for  252s at: Test4GBStoredFields.test

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:55:53, stalled for 72.1s at: TestIndexWriterExceptions.testTooManyTokens

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:55:53, stalled for 64.6s at: TestSearcherManager.testSearcherManager

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:56:29, stalled for  312s at: Test4GBStoredFields.test

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:56:53, stalled for  132s at: TestIndexWriterExceptions.testTooManyTokens

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:56:53, stalled for  125s at: TestSearcherManager.testSearcherManager

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T01:57:29, stalled for  372s at: Test4GBStoredFields.test

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:57:53, stalled for  192s at: TestIndexWriterExceptions.testTooManyTokens

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:57:53, stalled for  185s at: TestSearcherManager.testSearcherManager

   [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields

   [junit4] Completed [198/437] on J2 in 426.43s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.bkd.TestDocIdsWriter

   [junit4] Completed [199/437] on J2 in 8.17s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDirectoryReader

   [junit4] Completed [200/437] on J2 in 2.64s, 24 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:58:53, stalled for  245s at: TestSearcherManager.testSearcherManager

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:58:53, stalled for  252s at: TestIndexWriterExceptions.testTooManyTokens

   [junit4] Suite: org.apache.lucene.geo.TestGeoUtils

   [junit4] Completed [201/437] on J2 in 42.77s, 6 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T01:59:53, stalled for  305s at: TestSearcherManager.testSearcherManager

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T01:59:53, stalled for  312s at: TestIndexWriterExceptions.testTooManyTokens

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterExceptions

   [junit4] Completed [202/437] on J1 in 356.69s, 36 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDeletionPolicy

   [junit4] Completed [203/437] on J1 in 5.51s, 6 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:00:29, stalled for 71.8s at: TestNRTThreads.testNRTThreads

   [junit4] Suite: org.apache.lucene.index.TestFlushByRamOrCountsPolicy

   [junit4] Completed [204/437] on J1 in 6.59s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSearcherManager

   [junit4] Completed [205/437] on J0 in 351.30s, 10 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBoolean2

   [junit4] Completed [206/437] on J0 in 5.24s, 10 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestPersistentSnapshotDeletionPolicy

   [junit4] Completed [207/437] on J0 in 1.74s, 13 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDuelingCodecs

   [junit4] Completed [208/437] on J1 in 15.94s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestAtomicUpdate

   [junit4] Completed [209/437] on J0 in 5.99s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNorms

   [junit4] Completed [210/437] on J1 in 5.04s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDirectoryReaderReopen

   [junit4] Completed [211/437] on J1 in 2.70s, 14 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestTimeLimitingCollector

   [junit4] Completed [212/437] on J1 in 5.99s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.compressing.TestHighCompressionMode

   [junit4] Completed [213/437] on J1 in 2.86s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.standard.TestStandardAnalyzer

   [junit4] Completed [214/437] on J0 in 19.17s, 34 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.automaton.TestMinimize

   [junit4] Completed [215/437] on J0 in 3.31s, 3 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:01:29, stalled for  132s at: TestNRTThreads.testNRTThreads

   [junit4] Suite: org.apache.lucene.search.TestTopDocsMerge

   [junit4] Completed [216/437] on J1 in 28.32s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSnapshotDeletionPolicy

   [junit4] Completed [217/437] on J1 in 1.54s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestRollingUpdates

   [junit4] Completed [218/437] on J0 in 22.88s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterForceMerge

   [junit4] Completed [219/437] on J0 in 4.43s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.automaton.TestLevenshteinAutomata

   [junit4] Completed [220/437] on J0 in 1.71s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestBufferedIndexInput

   [junit4] Completed [221/437] on J0 in 0.38s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTieredMergePolicy

   [junit4] Completed [222/437] on J1 in 10.89s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestWeakIdentityMap

   [junit4] Completed [223/437] on J1 in 16.41s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSimpleExplanations

   [junit4] Completed [224/437] on J1 in 1.18s, 70 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestLockFactory

   [junit4] Completed [225/437] on J1 in 0.04s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSegmentMerger

   [junit4] Completed [226/437] on J1 in 0.74s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestRegexpRandom2

   [junit4] Completed [227/437] on J1 in 4.04s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.compressing.TestFastDecompressionMode

   [junit4] Completed [228/437] on J1 in 3.70s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterNRTIsCurrent

   [junit4] Completed [229/437] on J1 in 6.22s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestPagedBytes

   [junit4] Completed [230/437] on J1 in 1.89s, 4 tests, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTransactions

   [junit4] Completed [231/437] on J1 in 2.85s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestCrash

   [junit4] Completed [232/437] on J1 in 1.16s, 5 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:02:29, stalled for  192s at: TestNRTThreads.testNRTThreads

   [junit4] Suite: org.apache.lucene.index.TestSegmentTermDocs

   [junit4] Completed [233/437] on J1 in 0.30s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.perfield.TestPerFieldPostingsFormat2

   [junit4] Completed [234/437] on J1 in 4.90s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSort

   [junit4] Completed [235/437] on J1 in 0.27s, 25 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestToken

   [junit4] Completed [236/437] on J1 in 0.02s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSimpleSearchEquivalence

   [junit4] Completed [237/437] on J1 in 11.40s, 17 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestHugeRamFile

   [junit4] Completed [238/437] on J1 in 1.44s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBooleanMinShouldMatch

   [junit4] Completed [239/437] on J1 in 3.82s, 17 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:02:53, stalled for 67.7s at: TestTermsEnum.testVaryingTermsPerSegment

   [junit4] Suite: org.apache.lucene.index.TestTermsEnum2

   [junit4] Completed [240/437] on J1 in 4.34s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestLegacyNumericUtils

   [junit4] Completed [241/437] on J1 in 1.35s, 12 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.automaton.TestDeterminizeLexicon

   [junit4] Completed [242/437] on J1 in 2.45s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:03:29, stalled for  252s at: TestNRTThreads.testNRTThreads

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:03:53, stalled for  128s at: TestTermsEnum.testVaryingTermsPerSegment

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnDiskFull

   [junit4] Completed [243/437] on J1 in 52.62s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestPostingsOffsets

   [junit4] Completed [244/437] on J1 in 3.64s, 13 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMixedCodecs

   [junit4] Completed [245/437] on J1 in 3.19s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTransactionRollback

   [junit4] Completed [246/437] on J1 in 0.24s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestRAMDirectory

   [junit4] IGNOR/A 0.00s J1 | TestRAMDirectory.testFsyncDoesntCreateNewFiles

   [junit4]    > Assumption #1: test only works for FSDirectory subclasses

   [junit4] IGNOR/A 0.00s J1 | TestRAMDirectory.testPendingDeletions

   [junit4]    > Assumption #1: we can only install VirusCheckingFS on an FSDirectory

   [junit4] Completed [247/437] on J1 in 1.46s, 48 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiPhraseQuery

   [junit4] Completed [248/437] on J1 in 0.21s, 16 tests, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestLazyProxSkipping

   [junit4] Completed [249/437] on J1 in 0.41s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.automaton.TestDeterminism

   [junit4] Completed [250/437] on J1 in 1.24s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.TestMergeSchedulerExternal

   [junit4] Completed [251/437] on J1 in 0.96s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTermVectorsWriter

   [junit4] Completed [252/437] on J1 in 0.29s, 15 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestCodecs

   [junit4] Completed [253/437] on J1 in 0.30s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestParallelLeafReader

   [junit4] Completed [254/437] on J1 in 0.18s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestStressNRT

   [junit4] Completed [255/437] on J1 in 0.74s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexInput

   [junit4] Completed [256/437] on J1 in 1.00s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestDisjunctionMaxQuery

   [junit4] Completed [257/437] on J1 in 0.33s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestNearSpansOrdered

   [junit4] Completed [258/437] on J1 in 0.47s, 19 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMultiFields

   [junit4] Completed [259/437] on J1 in 2.78s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestScorerPerf

   [junit4] Completed [260/437] on J1 in 9.31s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiTermConstantScore

   [junit4] Completed [261/437] on J1 in 0.53s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestForTooMuchCloning

   [junit4] Completed [262/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestFuzzyQuery

   [junit4] Completed [263/437] on J1 in 0.17s, 10 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestWildcard

   [junit4] Completed [264/437] on J1 in 0.04s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.similarities.TestSimilarity2

   [junit4] Completed [265/437] on J1 in 0.22s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestFieldMaskingSpanQuery

   [junit4] IGNOR/A 0.00s J1 | TestFieldMaskingSpanQuery.testSimple2

   [junit4]    > Assumption #1: Broken scoring: LUCENE-3723

   [junit4] IGNOR/A 0.00s J1 | TestFieldMaskingSpanQuery.testSpans2

   [junit4]    > Assumption #1: Broken scoring: LUCENE-3723

   [junit4] Completed [266/437] on J1 in 0.15s, 11 tests, 2 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.TestExternalCodecs

   [junit4] Completed [267/437] on J1 in 0.15s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNoDeletionPolicy

   [junit4] Completed [268/437] on J1 in 0.08s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSimpleExplanationsOfNonMatches

   [junit4] Completed [269/437] on J1 in 0.05s, 70 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestPerSegmentDeletes

   [junit4] Completed [270/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiValuedNumericRangeQuery

   [junit4] Completed [271/437] on J1 in 1.23s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestPrefixInBooleanQuery

   [junit4] Completed [272/437] on J1 in 0.48s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestTopDocsCollector

   [junit4] Completed [273/437] on J1 in 0.03s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestUniqueTermCount

   [junit4] Completed [274/437] on J1 in 0.03s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestUnicodeUtil

   [junit4] Completed [275/437] on J1 in 0.69s, 5 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:04:29, stalled for  312s at: TestNRTThreads.testNRTThreads

   [junit4] Suite: org.apache.lucene.search.TestSortRandom

   [junit4] Completed [276/437] on J1 in 0.92s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestPrefixCodedTerms

   [junit4] Completed [277/437] on J1 in 0.23s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMaxTermFrequency

   [junit4] Completed [278/437] on J1 in 0.16s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestTopFieldCollector

   [junit4] Completed [279/437] on J1 in 0.12s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestDocValuesScoring

   [junit4] Completed [280/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestSentinelIntSet

   [junit4] Completed [281/437] on J1 in 0.10s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestWildcardRandom

   [junit4] Completed [282/437] on J1 in 0.14s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestDirectory

   [junit4] Completed [283/437] on J1 in 0.98s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestRegexpRandom

   [junit4] Completed [284/437] on J1 in 0.19s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMultiLevelSkipList

   [junit4] Completed [285/437] on J1 in 2.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestCheckIndex

   [junit4] Completed [286/437] on J1 in 2.17s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestPayloadsOnVectors

   [junit4] Completed [287/437] on J1 in 0.07s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanMultiTermQueryWrapper

   [junit4] Completed [288/437] on J1 in 0.13s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.TestSearchForDuplicates

   [junit4] Completed [289/437] on J1 in 0.37s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.automaton.TestCompiledAutomaton

   [junit4] Completed [290/437] on J1 in 0.25s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNRTThreads

   [junit4] Completed [291/437] on J2 in 321.22s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestConstantScoreQuery

   [junit4] Completed [292/437] on J1 in 0.02s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMatchAllDocsQuery

   [junit4] Completed [293/437] on J1 in 0.05s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTermsEnum

   [junit4] Completed [294/437] on J0 in 173.31s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestBinaryTerms

   [junit4] Completed [295/437] on J1 in 0.19s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSubScorerFreqs

   [junit4] Completed [296/437] on J0 in 0.13s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTerms

   [junit4] Completed [297/437] on J2 in 0.90s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestConjunctions

   [junit4] Completed [298/437] on J0 in 0.04s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiTermQueryRewrites

   [junit4] Completed [299/437] on J0 in 0.05s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestRecyclingIntBlockAllocator

   [junit4] Completed [300/437] on J1 in 0.73s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestDateSort

   [junit4] Completed [301/437] on J0 in 0.03s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanExplanationsOfNonMatches

   [junit4] Completed [302/437] on J1 in 0.04s, 30 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestVirtualMethod

   [junit4] Completed [303/437] on J0 in 0.03s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestRollingBuffer

   [junit4] Completed [304/437] on J1 in 0.11s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestNumericTokenStream

   [junit4] Completed [305/437] on J0 in 0.01s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.TestSearch

   [junit4] Completed [306/437] on J1 in 0.02s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestFilterIterator

   [junit4] Completed [307/437] on J0 in 0.02s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestCharsRef

   [junit4] Completed [308/437] on J1 in 0.03s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestPriorityQueue

   [junit4] Completed [309/437] on J0 in 0.20s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSameTokenSamePosition

   [junit4] Completed [310/437] on J1 in 0.03s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocCount

   [junit4] Completed [311/437] on J0 in 0.09s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.document.TestBinaryDocument

   [junit4] Completed [312/437] on J1 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestMmapDirectory

   [junit4] Completed [313/437] on J2 in 2.00s, 44 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestWordlistLoader

   [junit4] Completed [314/437] on J2 in 0.03s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestRecyclingByteBlockAllocator

   [junit4] Completed [315/437] on J0 in 0.13s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestRamUsageEstimator

   [junit4] IGNOR/A 0.00s J1 | TestRamUsageEstimator.testPrintValues

   [junit4]    > Assumption #1: Specify -Dtests.verbose=true to print constants of RamUsageEstimator.

   [junit4] Completed [316/437] on J1 in 0.03s, 5 tests, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanFirstQuery

   [junit4] Completed [317/437] on J0 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIsCurrent

   [junit4] Completed [318/437] on J1 in 0.03s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNoMergeScheduler

   [junit4] Completed [319/437] on J0 in 0.01s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestNamedSPILoader

   [junit4] Completed [320/437] on J1 in 0.01s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestBytesRef

   [junit4] Completed [321/437] on J0 in 0.01s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.TestAssertions

   [junit4] Completed [322/437] on J1 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestCharFilter

   [junit4] Completed [323/437] on J0 in 0.04s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestNot

   [junit4] Completed [324/437] on J1 in 0.04s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestCodecHoldsOpenFiles

   [junit4] Completed [325/437] on J1 in 0.04s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestReaderClosed

   [junit4] Completed [326/437] on J1 in 0.01s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.MultiCollectorTest

   [junit4] Completed [327/437] on J1 in 0.02s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestDocBoost

   [junit4] Completed [328/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestNGramPhraseQuery

   [junit4] Completed [329/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestScoreCachingWrappingScorer

   [junit4] Completed [330/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestIntsRef

   [junit4] Completed [331/437] on J1 in 0.02s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.tokenattributes.TestSimpleAttributeImpl

   [junit4] Completed [332/437] on J1 in 0.03s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTermdocPerf

   [junit4] Completed [333/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexCommit

   [junit4] Completed [334/437] on J1 in 0.06s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestIOUtils

   [junit4] Completed [335/437] on J1 in 0.18s, 18 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.Test2BPagedBytes

   [junit4] IGNOR/A 0.00s J1 | Test2BPagedBytes.test

   [junit4]    > Assumption #1: 'monster' test group is disabled (@Monster(value=You must increase heap to > 2 G to run this))

   [junit4] Completed [336/437] on J1 in 0.02s, 1 test, 1 skipped

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestCharArrayMap

   [junit4] Completed [337/437] on J1 in 0.26s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestCharacterUtils

   [junit4] Completed [338/437] on J1 in 0.04s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.TestReusableStringReader

   [junit4] Completed [339/437] on J1 in 0.13s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.tokenattributes.TestBytesRefAttImpl

   [junit4] Completed [340/437] on J1 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.tokenattributes.TestPackedTokenAttributeImpl

   [junit4] Completed [341/437] on J1 in 0.01s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOnJRECrash

   [junit4] Completed [342/437] on J0 in 4.64s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.TestCodecUtil

   [junit4] Completed [343/437] on J0 in 0.03s, 15 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.TestCodecLoadingDeadlock

   [junit4] Completed [344/437] on J1 in 0.26s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestForUtil

   [junit4] Completed [345/437] on J0 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50FieldInfoFormat

   [junit4] Completed [346/437] on J0 in 0.83s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestLucene50CompoundFormat

   [junit4] Completed [347/437] on J1 in 4.75s, 26 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestCollectionUtil

   [junit4] Completed [348/437] on J2 in 13.40s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.codecs.lucene53.TestLucene53NormsFormat

   [junit4] IGNOR/A 0.00s J0 | TestLucene53NormsFormat.testMergeStability

   [junit4]    > Assumption #1: The MockRandom PF randomizes content on the fly, so we can't check it

   [junit4] Completed [349/437] on J0 in 100.41s, 17 tests, 1 skipped

   [junit4] 

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T02:06:29, stalled for 60.1s at: TestLucene54DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields

   [junit4] Suite: org.apache.lucene.geo.TestGeoEncodingUtils

   [junit4] Completed [350/437] on J0 in 0.54s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestAllFilesHaveChecksumFooter

   [junit4] Completed [351/437] on J0 in 0.13s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:07:05, stalled for 64.3s at: TestPointQueries.testAllEqual

   [junit4] Suite: org.apache.lucene.search.TestPointQueries

   [junit4] Completed [352/437] on J2 in 140.43s, 46 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestOmitTf

   [junit4] Completed [353/437] on J2 in 0.48s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestFixedBitDocIdSet

   [junit4] Completed [354/437] on J2 in 0.91s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.document.TestField

   [junit4] Completed [355/437] on J2 in 0.09s, 27 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocumentWriter

   [junit4] Completed [356/437] on J2 in 0.15s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.packed.TestDirectMonotonic

   [junit4] Completed [357/437] on J2 in 2.52s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestReaderWrapperDVTypeCheck

   [junit4] Completed [358/437] on J2 in 0.03s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestSparseFixedBitSet

   [junit4] Completed [359/437] on J2 in 2.36s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestPayloads

   [junit4] Completed [360/437] on J2 in 0.13s, 7 tests

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:07:41, stalled for 69.6s at: TestBinaryDocValuesUpdates.testTonsOfUpdates

   [junit4] Suite: org.apache.lucene.codecs.lucene50.TestBlockPostingsFormat

   [junit4] Completed [361/437] on J2 in 41.09s, 26 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterDeleteByQuery

   [junit4] Completed [362/437] on J2 in 0.05s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:08:41, stalled for  130s at: TestBinaryDocValuesUpdates.testTonsOfUpdates

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:09:05, stalled for 62.2s at: Test2BPostings.test

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:09:41, stalled for  190s at: TestBinaryDocValuesUpdates.testTonsOfUpdates

   [junit4] Suite: org.apache.lucene.index.TestBinaryDocValuesUpdates

   [junit4] Completed [363/437] on J0 in 212.20s, 28 tests

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:10:05, stalled for  122s at: Test2BPostings.test

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T02:10:17, stalled for 69.7s at: TestLucene54DocValuesFormat.testSparseDocValuesVsStoredFields

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:11:05, stalled for  182s at: Test2BPostings.test

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T02:11:17, stalled for  130s at: TestLucene54DocValuesFormat.testSparseDocValuesVsStoredFields

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:11:41, stalled for 69.6s at: TestDemoParallelLeafReader.testRandomMultipleSchemaGens

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:12:05, stalled for  242s at: Test2BPostings.test

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:12:41, stalled for  130s at: TestDemoParallelLeafReader.testRandomMultipleSchemaGens

   [junit4] Suite: org.apache.lucene.codecs.lucene54.TestLucene54DocValuesFormat

   [junit4] Completed [364/437] on J1 in 491.02s, 109 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDocIDMerger

   [junit4] Completed [365/437] on J1 in 0.03s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestExceedMaxTermLength

   [junit4] Completed [366/437] on J1 in 0.02s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:13:05, stalled for  302s at: Test2BPostings.test

   [junit4] Suite: org.apache.lucene.index.TestIndexWriterOutOfFileDescriptors

   [junit4] Completed [367/437] on J1 in 7.18s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:13:41, stalled for  190s at: TestDemoParallelLeafReader.testRandomMultipleSchemaGens

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:14:05, stalled for  362s at: Test2BPostings.test

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T02:14:17, stalled for 65.5s at: TestIndexingSequenceNumbers.testStressConcurrentAddAndDeleteAndCommit

   [junit4] Suite: org.apache.lucene.index.Test2BPostings

   [junit4] Completed [368/437] on J2 in 413.87s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestPhrasePrefixQuery

   [junit4] Completed [369/437] on J2 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBlendedTermQuery

   [junit4] Completed [370/437] on J2 in 0.02s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestLongPostings

   [junit4] Completed [371/437] on J2 in 3.74s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMergePolicyWrapper

   [junit4] Completed [372/437] on J2 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestUpgradeIndexMergePolicy

   [junit4] Completed [373/437] on J2 in 0.08s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSumDocFreq

   [junit4] Completed [374/437] on J2 in 0.15s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.TestDemo

   [junit4] Completed [375/437] on J2 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTerm

   [junit4] Completed [376/437] on J2 in 0.02s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.util.TestInPlaceMergeSorter

   [junit4] Completed [377/437] on J2 in 0.10s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMaxPosition

   [junit4] Completed [378/437] on J2 in 0.02s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMergeRateLimiter

   [junit4] Completed [379/437] on J2 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestMultiTermsEnum

   [junit4] Completed [380/437] on J2 in 0.04s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestNRTReaderCleanup

   [junit4] Completed [381/437] on J2 in 0.01s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestOrdinalMap

   [junit4] Completed [382/437] on J2 in 0.76s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestReadOnlyIndex

   [junit4] Completed [383/437] on J2 in 0.14s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestSegmentInfos

   [junit4] Completed [384/437] on J2 in 0.03s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestStressDeletes

   [junit4] Completed [385/437] on J2 in 1.14s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J0 PID(14711@localhost): 2016-07-31T02:15:05, stalled for 63.5s at: TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField

   [junit4] Suite: org.apache.lucene.index.TestTermVectors

   [junit4] Completed [386/437] on J2 in 0.26s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestTryDelete

   [junit4] Completed [387/437] on J2 in 0.02s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.FuzzyTermOnShortTermsTest

   [junit4] Completed [388/437] on J2 in 0.18s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.index.TestDemoParallelLeafReader

   [junit4] Completed [389/437] on J0 in 303.70s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBooleanRewrites

   [junit4] Completed [390/437] on J0 in 0.66s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestBoostQuery

   [junit4] Completed [391/437] on J0 in 0.03s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestApproximationSearchEquivalence

   [junit4] Completed [392/437] on J2 in 7.45s, 10 tests

   [junit4] 

   [junit4] HEARTBEAT J1 PID(14712@localhost): 2016-07-31T02:15:17, stalled for  125s at: TestIndexingSequenceNumbers.testStressConcurrentAddAndDeleteAndCommit

   [junit4] Suite: org.apache.lucene.search.TestConjunctionDISI

   [junit4] Completed [393/437] on J0 in 10.19s, 5 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestDocValuesRewriteMethod

   [junit4] Completed [394/437] on J0 in 7.29s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestEarlyTermination

   [junit4] Completed [395/437] on J0 in 0.36s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestFieldValueQuery

   [junit4] Completed [396/437] on J0 in 7.49s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestFilterWeight

   [junit4] Completed [397/437] on J0 in 0.03s, 1 test

   [junit4] 

   [junit4] HEARTBEAT J2 PID(14713@localhost): 2016-07-31T02:16:17, stalled for 63.6s at: TestControlledRealTimeReopenThread.testControlledRealTimeReopenThread

   [junit4] Suite: org.apache.lucene.index.TestIndexingSequenceNumbers

   [junit4] Completed [398/437] on J1 in 215.57s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestMultiPhraseEnum

   [junit4] Completed [399/437] on J1 in 0.21s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestNeedsScores

   [junit4] Completed [400/437] on J1 in 0.05s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestQueryRescorer

   [junit4] Completed [401/437] on J1 in 1.62s, 6 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestReqExclBulkScorer

   [junit4] Completed [402/437] on J1 in 0.09s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSortedNumericSortField

   [junit4] Completed [403/437] on J1 in 0.24s, 9 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.TestSortedSetSelector

   [junit4] Completed [404/437] on J1 in 0.54s, 15 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestFilterSpans

   [junit4] Completed [405/437] on J1 in 0.22s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanCollection

   [junit4] Completed [406/437] on J1 in 0.10s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanNotQuery

   [junit4] Completed [407/437] on J1 in 0.08s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanOrQuery

   [junit4] Completed [408/437] on J1 in 0.04s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpanTermQuery

   [junit4] Completed [409/437] on J1 in 0.03s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.search.spans.TestSpansEnum

   [junit4] Completed [410/437] on J1 in 0.14s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestBufferedChecksum

   [junit4] Completed [411/437] on J1 in 0.06s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.store.TestNIOFSDirectory

   [junit4] Completed [412/437] on J1 in 3.77s, 44 tests

   [junit4] 


[...truncated too long message...]

ce/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/java/org/apache/lucene/collation/tokenattributes/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/java/org/apache/lucene/analysis/gl/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/java/org/apache/lucene/analysis/commongrams/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/java/org/apache/lucene/analysis/compound/package-info.class

     [copy] Copying 39 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/java


compile-test-framework:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


init:


compile-lucene-core:


compile-codecs:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


common.init:


compile-lucene-core:


init:


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


compile-core:


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


common.compile-core:


compile-core:


common.compile-test:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/test

    [javac] Compiling 276 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/test

    [javac] Note: Some input files use or override a deprecated API.

    [javac] Note: Recompile with -Xlint:deprecation for details.

     [copy] Copying 112 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/classes/test


install-junit4-taskdef:


validate:


resolve-groovy:


-init-totals:


-test:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/test

[loadresource] Do not set property tests.explicitclass as its length is 0.

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/common/test/temp

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/.caches/test-stats/analyzers-common

   [junit4] <JUnit4> says jolly good day! Master seed: 2CFAFCD9DBA77CCB

   [junit4] Executing 272 suites with 3 JVMs.

   [junit4] 

   [junit4] Started J1 PID(24365@localhost).

   [junit4] Started J2 PID(24364@localhost).

   [junit4] Started J0 PID(24366@localhost).

   [junit4] Suite: org.apache.lucene.analysis.tr.TestTurkishLowerCaseFilterFactory

   [junit4] Completed [1/272] on J0 in 0.54s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.in.TestIndicNormalizer

   [junit4] Completed [2/272] on J1 in 0.82s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.commongrams.TestCommonGramsQueryFilterFactory

   [junit4] Completed [3/272] on J1 in 0.13s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.standard.TestStandardFactories

   [junit4] Completed [4/272] on J2 in 1.19s, 11 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.core.TestStopFilterFactory

   [junit4] Completed [5/272] on J2 in 0.12s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.en.TestEnglishAnalyzer

   [junit4] Completed [6/272] on J0 in 0.82s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.no.TestNorwegianMinimalStemFilterFactory

   [junit4] Completed [7/272] on J0 in 0.15s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.da.TestDanishAnalyzer

   [junit4] Completed [8/272] on J1 in 0.91s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.util.TestCharArrayIterator

   [junit4] Completed [9/272] on J2 in 0.38s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.de.TestGermanLightStemFilterFactory

   [junit4] Completed [10/272] on J2 in 0.10s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.payloads.TokenOffsetPayloadTokenFilterTest

   [junit4] Completed [11/272] on J2 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.sv.TestSwedishAnalyzer

   [junit4] Completed [12/272] on J0 in 0.55s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.pattern.TestPatternTokenizerFactory

   [junit4] Completed [13/272] on J1 in 0.10s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.hunspell.TestFlagNum

   [junit4] Completed [14/272] on J2 in 0.22s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.hunspell.TestIgnore

   [junit4] Completed [15/272] on J0 in 0.18s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.eu.TestBasqueAnalyzer

   [junit4] Completed [16/272] on J0 in 0.41s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.el.TestGreekLowerCaseFilterFactory

   [junit4] Completed [17/272] on J0 in 0.07s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.fa.TestPersianNormalizationFilter

   [junit4] Completed [18/272] on J0 in 0.10s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.hunspell.TestCaseSensitive

   [junit4] Completed [19/272] on J0 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.hunspell.TestCondition

   [junit4] Completed [20/272] on J0 in 0.04s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.miscellaneous.TestKeywordRepeatFilter

   [junit4] Completed [21/272] on J0 in 0.04s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.de.TestGermanLightStemFilter

   [junit4] Completed [22/272] on J1 in 7.16s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.hunspell.TestHomonyms

   [junit4] Completed [23/272] on J1 in 0.11s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.reverse.TestReverseStringFilterFactory

   [junit4] Completed [24/272] on J1 in 0.05s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.lv.TestLatvianStemFilterFactory

   [junit4] Completed [25/272] on J1 in 0.06s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.pt.TestPortugueseMinimalStemFilterFactory

   [junit4] Completed [26/272] on J1 in 0.07s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.standard.TestUAX29URLEmailAnalyzer

   [junit4] Completed [27/272] on J1 in 0.55s, 32 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.bg.TestBulgarianAnalyzer

   [junit4] Completed [28/272] on J1 in 0.19s, 7 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.miscellaneous.DateRecognizerFilterTest

   [junit4] Completed [29/272] on J1 in 0.05s, 1 test

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.compound.TestHyphenationCompoundWordTokenFilterFactory

   [junit4] Completed [30/272] on J1 in 0.41s, 3 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.wikipedia.WikipediaTokenizerTest

   [junit4] Completed [31/272] on J0 in 12.17s, 8 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.hy.TestArmenianAnalyzer

   [junit4] Completed [32/272] on J0 in 0.43s, 4 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.fr.TestFrenchMinimalStemFilterFactory

   [junit4] Completed [33/272] on J0 in 0.05s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.el.TestGreekStemFilterFactory

   [junit4] Completed [34/272] on J0 in 0.04s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.es.TestSpanishLightStemFilterFactory

   [junit4] Completed [35/272] on J0 in 0.04s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.miscellaneous.TestTrimFilterFactory

   [junit4] Completed [36/272] on J0 in 0.04s, 2 tests

   [junit4] 

   [junit4] Suite: org.apache.lucene.analysis.cjk.TestCJKWidthFilter

   [junit4] Completed [37/272] on J0 in 0.41s, 4 tests

   [junit4] 

Slave went offline during the build

ERROR: Connection was broken: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Build step 'Invoke Ant' marked build as failure
ERROR: Step ‘Archive the artifacts’ failed: no workspace for Lucene-Solr-NightlyTests-master #1084
ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for Lucene-Solr-NightlyTests-master #1084
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
ERROR: lucene is offline; cannot locate latest1.8
ERROR: lucene is offline; cannot locate latest1.8





[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1103 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1103/

7 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([B265937124280B01]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexSorting

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([B265937124280B01]:0)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1471773756467,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_BlockTreeOrds_0.doc, _0_BlockTreeOrds_0.tio, _0_BlockTreeOrds_0.tipo, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_BlockTreeOrds_0.doc, _1_BlockTreeOrds_0.tio, _1_BlockTreeOrds_0.tipo, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_BlockTreeOrds_0.doc, _2_BlockTreeOrds_0.tio, _2_BlockTreeOrds_0.tipo, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_BlockTreeOrds_0.doc, _3_BlockTreeOrds_0.tio, _3_BlockTreeOrds_0.tipo, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_BlockTreeOrds_0.doc, _4_BlockTreeOrds_0.tio, _4_BlockTreeOrds_0.tipo, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_BlockTreeOrds_0.doc, _5_BlockTreeOrds_0.tio, _5_BlockTreeOrds_0.tipo, segments_2]}]> but was:<[{indexVersion=1471773756467,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_BlockTreeOrds_0.doc, _0_BlockTreeOrds_0.tio, _0_BlockTreeOrds_0.tipo, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_BlockTreeOrds_0.doc, _1_BlockTreeOrds_0.tio, _1_BlockTreeOrds_0.tipo, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_BlockTreeOrds_0.doc, _2_BlockTreeOrds_0.tio, _2_BlockTreeOrds_0.tipo, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_BlockTreeOrds_0.doc, _3_BlockTreeOrds_0.tio, _3_BlockTreeOrds_0.tipo, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_BlockTreeOrds_0.doc, _4_BlockTreeOrds_0.tio, _4_BlockTreeOrds_0.tipo, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_BlockTreeOrds_0.doc, _5_BlockTreeOrds_0.tio, _5_BlockTreeOrds_0.tipo, segments_2]}, {indexVersion=1471773756467,generation=3,filelist=[_6.fdt, _6.fdx, _6.fnm, _6.nvd, _6.nvm, _6.si, _6_BlockTreeOrds_0.doc, _6_BlockTreeOrds_0.tio, _6_BlockTreeOrds_0.tipo, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1471773756467,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_BlockTreeOrds_0.doc, _0_BlockTreeOrds_0.tio, _0_BlockTreeOrds_0.tipo, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_BlockTreeOrds_0.doc, _1_BlockTreeOrds_0.tio, _1_BlockTreeOrds_0.tipo, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_BlockTreeOrds_0.doc, _2_BlockTreeOrds_0.tio, _2_BlockTreeOrds_0.tipo, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_BlockTreeOrds_0.doc, _3_BlockTreeOrds_0.tio, _3_BlockTreeOrds_0.tipo, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_BlockTreeOrds_0.doc, _4_BlockTreeOrds_0.tio, _4_BlockTreeOrds_0.tipo, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_BlockTreeOrds_0.doc, _5_BlockTreeOrds_0.tio, _5_BlockTreeOrds_0.tipo, segments_2]}]> but was:<[{indexVersion=1471773756467,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_BlockTreeOrds_0.doc, _0_BlockTreeOrds_0.tio, _0_BlockTreeOrds_0.tipo, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_BlockTreeOrds_0.doc, _1_BlockTreeOrds_0.tio, _1_BlockTreeOrds_0.tipo, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_BlockTreeOrds_0.doc, _2_BlockTreeOrds_0.tio, _2_BlockTreeOrds_0.tipo, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_BlockTreeOrds_0.doc, _3_BlockTreeOrds_0.tio, _3_BlockTreeOrds_0.tipo, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_BlockTreeOrds_0.doc, _4_BlockTreeOrds_0.tio, _4_BlockTreeOrds_0.tipo, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_BlockTreeOrds_0.doc, _5_BlockTreeOrds_0.tio, _5_BlockTreeOrds_0.tipo, segments_2]}, {indexVersion=1471773756467,generation=3,filelist=[_6.fdt, _6.fdx, _6.fnm, _6.nvd, _6.nvm, _6.si, _6_BlockTreeOrds_0.doc, _6_BlockTreeOrds_0.tio, _6_BlockTreeOrds_0.tipo, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([62448CDB6B4D1906:479397EB1B051705]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1227)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([62448CDB6B4D1906]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=208, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=228, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=213, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=223, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=218, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=238, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=233, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=198, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=193, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=203, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=208, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=228, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=213, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=223, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=218, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=238, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=233, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=198, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=193, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=203, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([62448CDB6B4D1906]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=47296, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=47296, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:54146, https://127.0.0.1:38648, https://127.0.0.1:38216, https://127.0.0.1:43166, https://127.0.0.1:50396]
	at __randomizedtesting.SeedInfo.seed([62448CDB6B4D1906]:0)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:998)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:54146, https://127.0.0.1:38648, https://127.0.0.1:38216, https://127.0.0.1:43166, https://127.0.0.1:50396]
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:392)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:988)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:38648: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:365)
	... 7 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=70855, name=org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d918fb, state=RUNNABLE, group=TGRP-HdfsRestartWhileUpdatingTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=70855, name=org.apache.hadoop.util.JvmPauseMonitor$Monitor@1d918fb, state=RUNNABLE, group=TGRP-HdfsRestartWhileUpdatingTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded




Build Log:
[...truncated 1772 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> kol 21, 2016 10:01:26 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexSorting
   [junit4]   2> ==== jstack at approximately timeout time ====
   [junit4]   2> "TEST-TestIndexSorting.testRandom3-seed#[B265937124280B01]" ID=4048 RUNNABLE
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.document(CompressingStoredFieldsReader.java:576)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:583)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.visitDocument(AssertingStoredFieldsFormat.java:73)
   [junit4]   2> 	at org.apache.lucene.codecs.StoredFieldsWriter.merge(StoredFieldsWriter.java:130)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:200)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:89)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
   [junit4]   2> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2> 	- locked org.apache.lucene.index.SerialMergeScheduler@2d8bc58d
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4928)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4966)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4957)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:1265)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> "SUITE-TestIndexSorting-seed#[B265937124280B01]" ID=4047 RUNNABLE
   [junit4]   2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
   [junit4]   2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:673)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:64)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:414)
   [junit4]   2> 	- locked java.lang.Object@46fdb0d9
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:681)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:591)
   [junit4]   2> 
   [junit4]   2> "JUnit4-serializer-daemon" ID=9 TIMED_WAITING
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
   [junit4]   2> 
   [junit4]   2> "Signal Dispatcher" ID=4 RUNNABLE
   [junit4]   2> 
   [junit4]   2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@1cec0ccf
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@1cec0ccf
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
   [junit4]   2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
   [junit4]   2> 
   [junit4]   2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@3e70272c
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.Reference$Lock@3e70272c
   [junit4]   2> 	at java.lang.Object.wait(Object.java:502)
   [junit4]   2> 	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
   [junit4]   2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
   [junit4]   2> 
   [junit4]   2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@6cb6cfb6
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@6cb6cfb6
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:601)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:450)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:243)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:354)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:10)
   [junit4]   2> 
   [junit4]   2> ^^==============================================
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.method=testRandom3 -Dtests.seed=B265937124280B01 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Europe/Andorra -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7190s J2 | TestIndexSorting.testRandom3 <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B265937124280B01]:0)
   [junit4]   2> kol 21, 2016 10:01:26 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> kol 21, 2016 10:01:46 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestIndexSorting: 
   [junit4]   2>    1) Thread[id=4048, name=TEST-TestIndexSorting.testRandom3-seed#[B265937124280B01], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2>         at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.postings(SegmentTermsEnum.java:998)
   [junit4]   2>         at org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.postings(FilterLeafReader.java:221)
   [junit4]   2>         at org.apache.lucene.index.AssertingLeafReader$AssertingTermsEnum.postings(AssertingLeafReader.java:156)
   [junit4]   2>         at org.apache.lucene.index.MultiTermsEnum.postings(MultiTermsEnum.java:359)
   [junit4]   2>         at org.apache.lucene.index.MappedMultiFields$MappedMultiTermsEnum.postings(MappedMultiFields.java:127)
   [junit4]   2>         at org.apache.lucene.codecs.asserting.AssertingPostingsFormat$AssertingFieldsConsumer.write(AssertingPostingsFormat.java:189)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
   [junit4]   2>         at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4928)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4966)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4957)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]   2>         at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:1265)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> kol 21, 2016 10:01:46 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=4048, name=TEST-TestIndexSorting.testRandom3-seed#[B265937124280B01], state=RUNNABLE, group=TGRP-TestIndexSorting]
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexSorting_B265937124280B01-001
   [junit4]   2> Aug 21, 2016 8:01:46 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=ClassicSimilarity, locale=hr, timezone=Europe/Andorra
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=151785592,total=500170752
   [junit4]   2> NOTE: All tests run in this JVM: [TestFilterLeafReader, TestSimilarity, TestSpansEnum, TestToken, TestRollback, TestRegExp, TestBinaryDocument, TestUpgradeIndexMergePolicy, TestDirectMonotonic, TestNamedSPILoader, TestHighCompressionMode, TestFlushByRamOrCountsPolicy, TestBlockPostingsFormat, TestSearchWithThreads, TestTopDocsMerge, TestWildcardRandom, TestLucene50StoredFieldsFormat, TestRamUsageEstimator, TestPositionIncrement, TestCompiledAutomaton, TestFilterWeight, TestMultiTermsEnum, TestPointQueries, TestLazyProxSkipping, TestIndexWriterExceptions, TestSmallFloat, TestNoMergePolicy, TestFieldMaskingSpanQuery, TestMmapDirectory, TestDocBoost, TestIndexWriterDeleteByQuery, TestParallelCompositeReader, TestSegmentMerger, TestClassicSimilarity, TestParallelLeafReader, TestIndexWriterCommit, TestSingleInstanceLockFactory, TestSparseFixedBitSet, TestIndexWriterMaxDocs, TestPrefixRandom, Test2BPoints, TestRollingBuffer, TestDocValuesIndexing, TestMultiTermConstantScore, TestTransactions, TestSimpleSearchEquivalence, TestMSBRadixSorter, Test2BPostingsBytes, TestTermQuery, TestPerSegmentDeletes, TestWeakIdentityMap, TestLogMergePolicy, TestSegmentTermEnum, TestBlendedTermQuery, TestSameScoresWithThreads, TestSnapshotDeletionPolicy, TestRegexpQuery, TestTopFieldCollector, TestOrdinalMap, TestDocInverterPerFieldErrorInfo, TestStressIndexing, TestIOUtils, TestConjunctionDISI, TestSimpleExplanationsWithFillerDocs, TestDocument, TestTermsEnum, TestLucene62SegmentInfoFormat, TestSegmentTermDocs, TestBagOfPostings, Test2BPagedBytes, TestWordlistLoader, TestAllFilesCheckIndexHeader, TestFileSwitchDirectory, TestIndexWriter, TestIndexWriterReader, TestIndexWriterMerging, TestFSTs, TestPackedInts, TestDuelingCodecs, TestBoolean2, TestPersistentSnapshotDeletionPolicy, TestAtomicUpdate, TestNorms, TestSearchAfter, TestConcurrentMergeScheduler, TestNeverDelete, TestIndexWriterForceMerge, TestPerFieldPostingsFormat, TestArrayUtil, TestLevenshteinAutomata, TestByteSlices, TestBufferedIndexInput, TestDirectoryReader, TestPhraseQuery, TestCollectionUtil, TestBagOfPositions, TestOmitNorms, TestComplexExplanations, TestRegexpRandom2, TestNRTCachingDirectory, TestFastDecompressionMode, TestPerFieldPostingsFormat2, TestSort, TestMultiDocValues, TestFieldsReader, TestCustomSearcherSort, TestHugeRamFile, TestBooleanMinShouldMatch, TestDeterminizeLexicon, TestSpans, TestCustomNorms, TestPostingsOffsets, TestMixedCodecs, TestTransactionRollback, TestFlex, TestSpanExplanations, TestFastCompressionMode, TestStressIndexing2, TestDocsAndPositions, TestTermVectorsWriter, TestCodecs, TestCharTermAttributeImpl, TestOmitPositions, TestBytesRefHash, TestThreadedForceMerge, TestStressNRT, TestIndexInput, TestSloppyPhraseQuery2, TestForceMergeForever, TestNearSpansOrdered, TestDocumentsWriterDeleteQueue, TestDocumentWriter, TestScorerPerf, TestLiveFieldValues, TestForTooMuchCloning, TestSpanSearchEquivalence, TestComplexExplanationsOfNonMatches, TestAutomatonQuery, TestDocValuesScoring, TestSentinelIntSet, TestPayloadsOnVectors, TestSpanMultiTermQueryWrapper, TestIndexFileDeleter, TestConjunctions, TestField, TestSpanExplanationsOfNonMatches, TestFilterIterator, Test2BPostings, TestGeoEncodingUtils, TestGeoUtils, TestDocIDMerger, TestDocValues, TestExceedMaxTermLength, TestFieldReuse, TestFilterDirectoryReader, TestIndexSorting]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexSorting -Dtests.seed=B265937124280B01 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Europe/Andorra -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestIndexSorting (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B265937124280B01]:0)
   [junit4] Completed [432/432 (1!)] on J2 in 7220.17s, 22 tests, 2 errors <<< FAILURES!

[...truncated 9466 lines...]
   [junit4] Suite: org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_62448CDB6B4D1906-001/init-core-data-001
   [junit4]   2> 42139 INFO  (SUITE-ConcurrentDeleteAndCreateCollectionTest-seed#[62448CDB6B4D1906]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 42142 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testConcurrentCreateAndDeleteDoesNotFail
   [junit4]   2> 42142 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 42143 INFO  (Thread-32) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 42143 INFO  (Thread-32) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 42242 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.ZkTestServer start zk server on port:58292
   [junit4]   2> 42242 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42243 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42250 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1a138224 name:ZooKeeperConnection Watcher:127.0.0.1:58292 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42250 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42250 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42250 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 42268 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/clusterprops.json
   [junit4]   2> 42344 INFO  (jetty-launcher-30-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 42346 INFO  (jetty-launcher-30-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@45cc7b63{/solr,null,AVAILABLE}
   [junit4]   2> 42542 INFO  (jetty-launcher-30-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@521d1791{SSL,[ssl, http/1.1]}{127.0.0.1:55234}
   [junit4]   2> 42542 INFO  (jetty-launcher-30-thread-1) [    ] o.e.j.s.Server Started @44799ms
   [junit4]   2> 42542 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55234}
   [junit4]   2> 42542 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@5c647e05
   [junit4]   2> 42542 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_62448CDB6B4D1906-001/tempDir-001/node1'
   [junit4]   2> 42542 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 42542 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 42543 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42545 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42547 INFO  (zkCallback-32-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@14ae5279 name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42547 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42548 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42548 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 42559 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_62448CDB6B4D1906-001/tempDir-001/node1/.
   [junit4]   2> 42559 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 1369998250
   [junit4]   2> 42559 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_62448CDB6B4D1906-001/tempDir-001/node1]
   [junit4]   2> 42560 WARN  (jetty-launcher-30-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_62448CDB6B4D1906-001/tempDir-001/node1/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_62448CDB6B4D1906-001/tempDir-001/node1/lib
   [junit4]   2> 42560 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 42574 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 42574 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 42574 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 42574 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58292/solr
   [junit4]   2> 42574 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 42575 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42575 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42577 INFO  (zkCallback-35-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3eeec7c6 name:ZooKeeperConnection Watcher:127.0.0.1:58292 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42577 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42578 INFO  (jetty-launcher-30-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42579 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$70/100602415@7863fb5a
   [junit4]   2> 42580 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42585 INFO  (zkCallback-36-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@23a5fa34 name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42585 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42591 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 42593 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 42596 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 42598 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 42600 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 42601 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 42603 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 42604 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 42605 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 42606 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 42607 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 42608 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 42608 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 42614 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 42615 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 42617 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 42617 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96454126515191811-127.0.0.1:55234_solr-n_0000000000
   [junit4]   2> 42618 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55234_solr
   [junit4]   2> 42618 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 42619 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.Overseer Overseer (id=96454126515191811-127.0.0.1:55234_solr-n_0000000000) starting
   [junit4]   2> 42621 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 42627 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 42628 INFO  (OverseerStateUpdate-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 42628 INFO  (OverseerCollectionConfigSetProcessor-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 42635 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55234_solr
   [junit4]   2> 42635 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:55234_solr
   [junit4]   2> 42636 INFO  (OverseerStateUpdate-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 42638 INFO  (zkCallback-36-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 42640 INFO  (zkCallback-36-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 42662 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 42662 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 42663 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_62448CDB6B4D1906-001/tempDir-001/node1/.
   [junit4]   2> 42663 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 42667 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1
   [junit4]   2> 42667 INFO  (jetty-launcher-30-thread-1) [n:127.0.0.1:55234_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 42667 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42672 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42685 INFO  (zkCallback-38-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@503e40 name:ZooKeeperConnection Watcher:127.0.0.1:58292 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42685 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42686 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42689 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42690 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42692 INFO  (zkCallback-40-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@486ec0ba name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42692 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42693 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42693 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection0/schema.xml
   [junit4]   2> 42698 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection0/solrconfig.xml
   [junit4]   2> 42703 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42707 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42709 INFO  (zkCallback-41-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@10c8a9d name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42709 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42709 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42709 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection1/schema.xml
   [junit4]   2> 42713 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 42716 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42717 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42726 INFO  (zkCallback-42-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@641917f1 name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42726 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42727 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42727 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection2/schema.xml
   [junit4]   2> 42729 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection2/solrconfig.xml
   [junit4]   2> 42739 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42740 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42747 INFO  (zkCallback-43-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@10ef505c name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42747 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42748 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42748 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection3/schema.xml
   [junit4]   2> 42752 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection3/solrconfig.xml
   [junit4]   2> 42761 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42763 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42764 INFO  (zkCallback-44-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7bf199db name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42764 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42765 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42765 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection4/schema.xml
   [junit4]   2> 42767 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection4/solrconfig.xml
   [junit4]   2> 42774 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42774 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42776 INFO  (zkCallback-45-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7703e67 name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42776 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42776 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42776 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection5/schema.xml
   [junit4]   2> 42779 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection5/solrconfig.xml
   [junit4]   2> 42785 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42786 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42798 INFO  (zkCallback-46-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7953a79f name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42798 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42798 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42798 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection6/schema.xml
   [junit4]   2> 42808 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection6/solrconfig.xml
   [junit4]   2> 42818 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42823 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42829 INFO  (zkCallback-47-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2099262 name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42829 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42830 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42830 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection7/schema.xml
   [junit4]   2> 42834 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection7/solrconfig.xml
   [junit4]   2> 42845 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42845 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42848 INFO  (zkCallback-48-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7f5cbaf3 name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42848 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42848 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42848 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection8/schema.xml
   [junit4]   2> 42852 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection8/solrconfig.xml
   [junit4]   2> 42857 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 42858 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 42860 INFO  (zkCallback-49-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@62ecb847 name:ZooKeeperConnection Watcher:127.0.0.1:58292/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 42860 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 42861 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 42861 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection9/schema.xml
   [junit4]   2> 42864 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail-seed#[62448CDB6B4D1906]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/collection9/solrconfig.xml
   [junit4]   2> 43158 INFO  (qtp343707366-162) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection6&name=collection6&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43164 INFO  (qtp343707366-245) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection2&name=collection2&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43165 INFO  (qtp343707366-243) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection3&name=collection3&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43168 INFO  (qtp343707366-241) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection1&name=collection1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43169 INFO  (qtp343707366-168) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection4&name=collection4&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43169 INFO  (qtp343707366-244) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection9&name=collection9&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43170 INFO  (OverseerCollectionConfigSetProcessor-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>   "name":"collection6",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection6",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43170 INFO  (qtp343707366-161) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection8&name=collection8&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43170 INFO  (qtp343707366-163) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection5&name=collection5&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43171 INFO  (qtp343707366-242) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection0&name=collection0&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43179 INFO  (qtp343707366-166) [n:127.0.0.1:55234_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=collection7&name=collection7&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 43179 INFO  (OverseerThreadFactory-66-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection6",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection6",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43179 INFO  (OverseerThreadFactory-66-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection6
   [junit4]   2> 43182 INFO  (OverseerThreadFactory-66-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/collection6 
   [junit4]   2> 43183 INFO  (OverseerThreadFactory-66-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection6
   [junit4]   2> 43184 INFO  (zkCallback-36-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 43185 INFO  (OverseerCollectionConfigSetProcessor-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2>   "name":"collection2",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection2",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43191 INFO  (OverseerCollectionConfigSetProcessor-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000003 message:{
   [junit4]   2>   "name":"collection3",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection3",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43191 INFO  (OverseerThreadFactory-66-thread-2-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection2",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection2",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43191 INFO  (OverseerThreadFactory-66-thread-2-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection2
   [junit4]   2> 43193 INFO  (OverseerCollectionConfigSetProcessor-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000004 message:{
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43193 INFO  (OverseerStateUpdate-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"collection6",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection6",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 43193 INFO  (OverseerStateUpdate-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection6
   [junit4]   2> 43195 INFO  (OverseerThreadFactory-66-thread-2-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/collection2 
   [junit4]   2> 43195 INFO  (OverseerStateUpdate-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection6/state.json
   [junit4]   2> 43195 INFO  (OverseerThreadFactory-66-thread-2-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection2
   [junit4]   2> 43197 INFO  (zkCallback-36-thread-1-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 43200 INFO  (OverseerThreadFactory-66-thread-3-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection3",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection3",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43200 INFO  (OverseerThreadFactory-66-thread-3-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection3
   [junit4]   2> 43200 INFO  (OverseerCollectionConfigSetProcessor-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000005 message:{
   [junit4]   2>   "name":"collection4",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection4",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43201 INFO  (OverseerThreadFactory-66-thread-3-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/collection3 
   [junit4]   2> 43202 INFO  (OverseerThreadFactory-66-thread-4-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43202 INFO  (OverseerThreadFactory-66-thread-4-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 43202 INFO  (OverseerCollectionConfigSetProcessor-96454126515191811-127.0.0.1:55234_solr-n_0000000000) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000006 message:{
   [junit4]   2>   "name":"collection9",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection9",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43202 INFO  (OverseerThreadFactory-66-thread-3-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection3
   [junit4]   2> 43202 INFO  (OverseerThreadFactory-66-thread-4-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/collection1 
   [junit4]   2> 43203 INFO  (OverseerThreadFactory-66-thread-5-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection4",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection4",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 43203 INFO  (OverseerThreadFactory-66-thread-4-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 43203 INFO  (OverseerThreadFactory-66-thread-5-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection4
   [junit4]   2> 43203 INFO  (OverseerThreadFactory-66-thread-6-processing-n:127.0.0.1:55234_solr) [n:127.0.0.1:55234_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection9",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"collection9",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "

[...truncated too long message...]

pdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
   [junit4]   2> 	at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:97)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:179)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
   [junit4]   2> 	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:274)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
   [junit4]   2> 	at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:239)
   [junit4]   2> 	at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:157)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:186)
   [junit4]   2> 	at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:107)
   [junit4]   2> 	at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
   [junit4]   2> 	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
   [junit4]   2> 	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:653)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:518)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 4829969 WARN  (SUITE-HdfsRestartWhileUpdatingTest-seed#[62448CDB6B4D1906]-worker) [    ] o.a.s.SolrTestCaseJ4 Some resources were not closed, shutdown, or released. This has been ignored due to the SuppressObjectReleaseTracker annotation, trying to close them now.
   [junit4]   2> Aug 21, 2016 11:12:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 36 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarity(queryNorm=false): {}, locale=no-NO, timezone=Asia/Colombo
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=2,free=68960216,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedFacetPivotLongTailTest, QueryResultKeyTest, TestManagedStopFilterFactory, LukeRequestHandlerTest, ConfigSetsAPITest, TestSolrQueryParserDefaultOperatorResource, JavabinLoaderTest, ParsingFieldUpdateProcessorsTest, ZkStateReaderTest, NoCacheHeaderTest, TestSearchPerf, TestFreeTextSuggestions, TestWriterPerf, TestHdfsCloudBackupRestore, CdcrReplicationHandlerTest, TestIndexingPerformance, TestSolrIndexConfig, TestDefaultSearchFieldResource, TestMiniSolrCloudClusterBase, TestManagedSchemaAPI, OverseerTest, SuggestComponentContextFilterQueryTest, BooleanFieldTest, TestSuggestSpellingConverter, DirectSolrConnectionTest, XmlUpdateRequestHandlerTest, HdfsWriteToMultipleCollectionsTest, CursorMarkTest, TestDistribIDF, BaseCdcrDistributedZkTest, WrapperMergePolicyFactoryTest, TestExtendedDismaxParser, ShowFileRequestHandlerTest, TestPHPSerializedResponseWriter, TestStressReorder, TestFieldSortValues, ClusterStateTest, TestMissingGroups, RestartWhileUpdatingTest, TestSha256AuthenticationProvider, FieldMutatingUpdateProcessorTest, TestBulkSchemaAPI, TestOrdValues, TestBulkSchemaConcurrent, OverseerStatusTest, LeaderElectionTest, DocValuesMultiTest, TestDistribDocBasedVersion, GraphQueryTest, TestOnReconnectListenerSupport, HdfsThreadLeakTest, TestSizeLimitedDistributedMap, LeaderInitiatedRecoveryOnCommitTest, UnloadDistributedZkTest, SaslZkACLProviderTest, TestFastWriter, DeleteStatusTest, TestObjectReleaseTracker, BadIndexSchemaTest, CurrencyFieldXmlFileTest, TestSolrJ, HdfsBasicDistributedZkTest, TestLocalFSCloudBackupRestore, TestCSVResponseWriter, TestJsonRequest, TestSchemaSimilarityResource, TolerantUpdateProcessorTest, SmileWriterTest, HdfsCollectionsAPIDistributedZkTest, TestZkChroot, TestJmxIntegration, RequestHandlersTest, ForceLeaderTest, RemoteQueryErrorTest, TestPerFieldSimilarity, ScriptEngineTest, InfoHandlerTest, TestSolrXml, HighlighterConfigTest, TestJettySolrRunner, TestConfigSetImmutable, TestLRUStatsCache, HdfsLockFactoryTest, TestTestInjection, HLLSerializationTest, TestShardHandlerFactory, SampleTest, SortSpecParsingTest, TestFieldCacheReopen, TestReversedWildcardFilterFactory, StatsComponentTest, TestCloudDeleteByQuery, TestSolr4Spatial2, TestConfigSetsAPIExclusivity, SuggesterTSTTest, AddSchemaFieldsUpdateProcessorFactoryTest, FieldAnalysisRequestHandlerTest, TestCharFilters, DocValuesMissingTest, TestInitParams, RecoveryZkTest, TestSQLHandler, TestExceedMaxTermLength, BasicDistributedZk2Test, SyncSliceTest, OpenCloseCoreStressTest, TestRecovery, TermVectorComponentDistributedTest, TestRealTimeGet, TestReload, HardAutoCommitTest, TestRangeQuery, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, CollectionReloadTest, CollectionsAPISolrJTest, DeleteNodeTest, DeleteShardTest, DistribCursorPagingTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerTaskQueueTest, RecoveryAfterSoftCommitTest, ShardSplitTest, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsRestartWhileUpdatingTest]
   [junit4] Completed [510/629 (4!)] on J2 in 117.08s, 1 test, 1 error <<< FAILURES!

[...truncated 366 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160821_095206_246.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid5566.hprof ...
   [junit4] Heap dump file created [689726402 bytes in 4.119 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 11034 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:715: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid5566.hprof

Total time: 361 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1102 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1102/

7 tests failed.
FAILED:  org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterRestart

Error Message:
Timeout while trying to assert number of documents @ source_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ source_collection
	at __randomizedtesting.SeedInfo.seed([1E8B0CE805D14DC0:4296EB69CBF1F9F6]:0)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:273)
	at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterRestart(CdcrReplicationDistributedZkTest.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: expected:<110> but was:<106>
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:264)
	... 42 more


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=67515, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=67515, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:53069, https://127.0.0.1:59503, https://127.0.0.1:32995, https://127.0.0.1:48072, https://127.0.0.1:52174]
	at __randomizedtesting.SeedInfo.seed([1E8B0CE805D14DC0]:0)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:998)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:53069, https://127.0.0.1:59503, https://127.0.0.1:32995, https://127.0.0.1:48072, https://127.0.0.1:52174]
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:392)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:988)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:48072
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:365)
	... 7 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 11 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at sun.security.ssl.InputRecord.read(InputRecord.java:505)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	... 27 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([1E8B0CE805D14DC0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=77019, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=77034, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=77014, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=77039, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=77004, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=77029, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=77044, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=77009, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=76999, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=77024, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=77019, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=77034, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=77014, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=77039, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=77004, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=77029, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=77044, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=77009, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=76999, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=77024, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([1E8B0CE805D14DC0]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:49183: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000096

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:49183: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000096
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.deleteCollectionWithDownNodes(CollectionsAPIDistributedZkTest.java:345)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:186)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest.test

Error Message:
KeeperErrorCode = ConnectionLoss for /

Stack Trace:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)
	at org.apache.solr.common.cloud.SolrZkClient.printLayout(SolrZkClient.java:585)
	at org.apache.solr.common.cloud.SolrZkClient.printLayoutToStdOut(SolrZkClient.java:626)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:181)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:862)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1418)
	at org.apache.solr.cloud.RestartWhileUpdatingTest.test(RestartWhileUpdatingTest.java:144)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test

Error Message:
IOException occured when talking to server at: https://127.0.0.1:45110//collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:45110//collection1
	at __randomizedtesting.SeedInfo.seed([1E8B0CE805D14DC0:96DF3332AB2D2038]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test(DistributedFacetPivotLongTailTest.java:94)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:209)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 48 more




Build Log:
[...truncated 11322 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/init-core-data-001
   [junit4]   2> 102185 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
   [junit4]   2> 102186 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 102186 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 102745 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 103292 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 103412 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 103427 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 103466 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 103504 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_57050_hdfs____.wmv97x/webapp
   [junit4]   2> 104414 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:57050
   [junit4]   2> 105520 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 105548 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 105555 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_50792_datanode____.gywqdg/webapp
   [junit4]   2> 105878 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:50792
   [junit4]   2> 106357 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 106362 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 106376 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40932_datanode____.acnirh/webapp
   [junit4]   2> 106724 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[1E8B0CE805D14DC0]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40932
   [junit4]   2> 107518 INFO  (IPC Server handler 7 on 47327) [    ] BlockStateChange BLOCK* processReport: from storage DS-7b3480a7-e6cc-4eb0-adaa-47830c4ec754 node DatanodeRegistration(127.0.0.1:41873, datanodeUuid=a55859a6-d36c-4cec-8872-060daaec5d21, infoPort=50877, infoSecurePort=0, ipcPort=48582, storageInfo=lv=-56;cid=testClusterID;nsid=1759858899;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 107518 INFO  (IPC Server handler 6 on 47327) [    ] BlockStateChange BLOCK* processReport: from storage DS-e2f746e2-4f68-4f65-a364-eaa7357e31f1 node DatanodeRegistration(127.0.0.1:51898, datanodeUuid=ae802aa1-468e-4cfa-b7dc-36e34d23943f, infoPort=48464, infoSecurePort=0, ipcPort=47775, storageInfo=lv=-56;cid=testClusterID;nsid=1759858899;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs
   [junit4]   2> 107519 INFO  (IPC Server handler 7 on 47327) [    ] BlockStateChange BLOCK* processReport: from storage DS-f03b10cf-89aa-4adc-b1ff-800b8f9e2426 node DatanodeRegistration(127.0.0.1:41873, datanodeUuid=a55859a6-d36c-4cec-8872-060daaec5d21, infoPort=50877, infoSecurePort=0, ipcPort=48582, storageInfo=lv=-56;cid=testClusterID;nsid=1759858899;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 107519 INFO  (IPC Server handler 6 on 47327) [    ] BlockStateChange BLOCK* processReport: from storage DS-e1474641-f58e-4782-a29d-8e5764acd9ab node DatanodeRegistration(127.0.0.1:51898, datanodeUuid=ae802aa1-468e-4cfa-b7dc-36e34d23943f, infoPort=48464, infoSecurePort=0, ipcPort=47775, storageInfo=lv=-56;cid=testClusterID;nsid=1759858899;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 107856 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 107856 INFO  (Thread-265) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 107856 INFO  (Thread-265) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 107956 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.ZkTestServer start zk server on port:53444
   [junit4]   2> 107956 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 107965 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 107989 INFO  (zkCallback-126-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2fb12867 name:ZooKeeperConnection Watcher:127.0.0.1:53444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 107989 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 107990 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 107990 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 108004 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 108007 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 108026 INFO  (zkCallback-127-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7104427a name:ZooKeeperConnection Watcher:127.0.0.1:53444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108029 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 108029 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 108029 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 108033 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 108034 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 108035 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 108036 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 108037 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 108039 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 108039 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 108041 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 108041 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 108043 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 108043 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 108045 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 108046 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 108047 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 108047 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 108051 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 108051 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 108052 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 108053 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 108054 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 108054 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 108055 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 108055 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 108056 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 108056 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 108070 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 108087 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 108093 INFO  (zkCallback-128-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7099edd1 name:ZooKeeperConnection Watcher:127.0.0.1:53444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108093 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 108093 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 108093 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 108096 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 108101 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 108110 INFO  (zkCallback-129-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@cab72cc name:ZooKeeperConnection Watcher:127.0.0.1:53444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108110 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 108110 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 108110 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 108110 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 108127 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 108127 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 108129 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 108129 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 108130 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 108130 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 108131 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 108131 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 108132 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 108132 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 108137 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 108137 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 108145 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 108145 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 108153 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 108153 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 108164 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 108164 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 108165 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 108165 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 108166 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 108166 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 108262 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/cores/collection1
   [junit4]   2> 108263 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 108266 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@705206c4{/,null,AVAILABLE}
   [junit4]   2> 108267 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.e.j.s.ServerConnector Started ServerConnector@374b46c3{HTTP/1.1,[http/1.1]}{127.0.0.1:34382}
   [junit4]   2> 108267 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.e.j.s.Server Started @110595ms
   [junit4]   2> 108267 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:47327/hdfs__localhost_47327__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001_tempDir-002_control_data, hostContext=/, hostPort=34382, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/cores}
   [junit4]   2> 108267 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@5c647e05
   [junit4]   2> 108268 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001'
   [junit4]   2> 108268 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 108268 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 108268 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 108268 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 108277 INFO  (zkCallback-130-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@13aed8a name:ZooKeeperConnection Watcher:127.0.0.1:53444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108277 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 108277 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 108279 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 108279 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/solr.xml
   [junit4]   2> 108285 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/cores
   [junit4]   2> 108285 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.CoreContainer New CoreContainer 1736612840
   [junit4]   2> 108285 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001]
   [junit4]   2> 108286 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/lib
   [junit4]   2> 108286 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 108303 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 108303 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 108303 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 108303 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53444/solr
   [junit4]   2> 108303 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 108303 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 108304 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 108317 INFO  (zkCallback-133-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5f64c794 name:ZooKeeperConnection Watcher:127.0.0.1:53444 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108322 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 108322 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 108329 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$12/1060417289@6acde4d7
   [junit4]   2> 108346 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 108350 INFO  (zkCallback-134-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2f80752e name:ZooKeeperConnection Watcher:127.0.0.1:53444/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 108350 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 108352 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 108354 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 108356 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 108357 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 108358 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 108359 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 108360 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 108361 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 108362 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 108362 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 108363 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 108363 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 108365 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 108369 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 108370 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 108370 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96447363607560198-127.0.0.1:34382_-n_0000000000
   [junit4]   2> 108370 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34382_
   [junit4]   2> 108370 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 108371 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.Overseer Overseer (id=96447363607560198-127.0.0.1:34382_-n_0000000000) starting
   [junit4]   2> 108373 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 108385 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 108390 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34382_
   [junit4]   2> 108390 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:34382_
   [junit4]   2> 108391 INFO  (OverseerCollectionConfigSetProcessor-96447363607560198-127.0.0.1:34382_-n_0000000000) [n:127.0.0.1:34382_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 108397 INFO  (OverseerStateUpdate-96447363607560198-127.0.0.1:34382_-n_0000000000) [n:127.0.0.1:34382_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 108398 INFO  (zkCallback-134-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 108409 INFO  (OverseerStateUpdate-96447363607560198-127.0.0.1:34382_-n_0000000000) [n:127.0.0.1:34382_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 108413 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 108413 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 108413 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/cores
   [junit4]   2> 108414 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 108414 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/cores/collection1
   [junit4]   2> 108414 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 108424 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 108424 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 108425 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2
   [junit4]   2> 108425 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[1E8B0CE805D14DC0]) [n:127.0.0.1:34382_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 108426 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 108426 INFO  (OverseerStateUpdate-96447363607560198-127.0.0.1:34382_-n_0000000000) [n:127.0.0.1:34382_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:34382",
   [junit4]   2>   "node_name":"127.0.0.1:34382_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 108426 INFO  (OverseerStateUpdate-96447363607560198-127.0.0.1:34382_-n_0000000000) [n:127.0.0.1:34382_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:34382",
   [junit4]   2>   "node_name":"127.0.0.1:34382_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 108426 INFO  (OverseerStateUpdate-96447363607560198-127.0.0.1:34382_-n_0000000000) [n:127.0.0.1:34382_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 108427 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 108428 INFO  (OverseerStateUpdate-96447363607560198-127.0.0.1:34382_-n_0000000000) [n:127.0.0.1:34382_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 108431 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 108431 INFO  (zkCallback-134-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 109432 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 109432 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 109433 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 109433 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 109433 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 109433 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/cores/collection1'
   [junit4]   2> 109433 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 109434 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 109435 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 109442 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 109445 WARN  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 109446 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 109455 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 109464 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 109557 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 109563 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 109564 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 109577 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 109579 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 109580 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 109581 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 109582 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2
   [junit4]   2> 109582 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 109582 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 109587 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:47327/solr_hdfs_home
   [junit4]   2> 109587 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 109587 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_1E8B0CE805D14DC0-001/control-001/cores/collection1], dataDir=[hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 109587 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6d76e2ce
   [junit4]   2> 109587 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 109632 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 109632 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.s.SolrSnapshotMetaDataManager Loading from snapshot metadata file...
   [junit4]   2> 109640 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 109658 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 109665 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 109680 WARN  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:47327/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 109680 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 109708 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:47327/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 109709 INFO  (coreLoadExecutor-270-thread-1-processing-n:127.0.0.1:34382_) [n:127.0.0.1:34382_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=24, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.0302734375, noCFSRatio=0.0]
   [junit4]   2> 109915 INFO  (IPC Server handler 1 on 47327) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51898 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e2f746e2-4f68-4f65-a364-eaa7357e31f1:NORMAL:127.0.0.1:51898|RBW], ReplicaUC[

[...truncated too long message...]

nterceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_102/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160820_051106_681.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160820_051106_681.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Forked process returned with error code: 11. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160820_051106_681.sysout
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1481)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:984: At least one slave process threw an exception, first: Forked process returned with error code: 11. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160820_051106_681.sysout

Total time: 296 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1101 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1101/

All tests passed

Build Log:
[...truncated 10336 lines...]
    [javac] Compiling 955 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/classes/java
    [javac] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/java/org/apache/solr/core/CoreSorter.java:24: warning: documentation comment not expected here
    [javac] package org.apache.solr.core;
    [javac] ^
    [javac] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/java/org/apache/solr/update/TransactionLog.java:174: error: variable o might not have been initialized
    [javac]           return;
    [javac]           ^
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 1 error

[...truncated 1 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:385: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:501: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1955: Compile failed; see the compiler error output for details.

Total time: 177 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1100 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1100/

11 tests failed.
FAILED:  org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([FC9E0D956239FF6A:1F9A5610C9223B2D]:0)
	at org.apache.lucene.util.fst.BytesStore.skipBytes(BytesStore.java:307)
	at org.apache.lucene.util.fst.FST.addNode(FST.java:792)
	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:126)
	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:214)
	at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:310)
	at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finishTerm(MemoryPostingsFormat.java:254)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.access$500(MemoryPostingsFormat.java:109)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:396)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4928)
	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4966)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4957)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
	at org.apache.lucene.index.TestDuelingCodecs.createRandomIndex(TestDuelingCodecs.java:139)
	at org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals(TestDuelingCodecsAtNight.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)


FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Captured an uncaught exception in thread: Thread[id=114722, name=Thread-105733, state=RUNNABLE, group=TGRP-TestDistributedSearch]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=114722, name=Thread-105733, state=RUNNABLE, group=TGRP-TestDistributedSearch]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:52576//collection1: java.lang.NullPointerException
	at org.apache.solr.search.grouping.distributed.responseprocessor.StoredFieldsShardResponseProcessor.process(StoredFieldsShardResponseProcessor.java:41)
	at org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:751)
	at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:734)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:422)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:653)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
	at org.apache.solr.TestDistributedSearch$1.run(TestDistributedSearch.java:1126)


FAILED:  org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterRestart

Error Message:
Timeout waiting for CDCR replication to complete @source_collection:shard1

Stack Trace:
java.lang.RuntimeException: Timeout waiting for CDCR replication to complete @source_collection:shard1
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C:B110EFE753FD345A]:0)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForReplicationToComplete(BaseCdcrDistributedZkTest.java:794)
	at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterRestart(CdcrReplicationDistributedZkTest.java:264)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterLeaderChange

Error Message:
Timeout waiting for CDCR replication to complete @source_collection:shard1

Stack Trace:
java.lang.RuntimeException: Timeout waiting for CDCR replication to complete @source_collection:shard1
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C:3FFD4485C372265E]:0)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForReplicationToComplete(BaseCdcrDistributedZkTest.java:794)
	at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterLeaderChange(CdcrReplicationDistributedZkTest.java:305)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:43125

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:43125
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.deleteCollectionWithDownNodes(CollectionsAPIDistributedZkTest.java:345)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:186)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 48 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsSyncSliceTest.test

Error Message:
There are still nodes recoverying - waited for 30 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C:655937BC3321ED94]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:182)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:862)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1418)
	at org.apache.solr.cloud.SyncSliceTest.test(SyncSliceTest.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Could not find collection:delete_data_dir

Stack Trace:
java.lang.AssertionError: Could not find collection:delete_data_dir
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C:655937BC3321ED94]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:134)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.join.BlockJoinFacetDistribTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest:     1) Thread[id=140219, name=OverseerHdfsCoreFailoverThread-96431830316154895-127.0.0.1:36657_solr-n_0000000002, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.search.join.BlockJoinFacetDistribTest: 
   1) Thread[id=140219, name=OverseerHdfsCoreFailoverThread-96431830316154895-127.0.0.1:36657_solr-n_0000000002, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.join.BlockJoinFacetDistribTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=140219, name=OverseerHdfsCoreFailoverThread-96431830316154895-127.0.0.1:36657_solr-n_0000000002, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=140219, name=OverseerHdfsCoreFailoverThread-96431830316154895-127.0.0.1:36657_solr-n_0000000002, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=110612, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=110602, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=110592, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=110607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=110637, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=110622, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=110627, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=110632, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=110617, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=110597, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=110612, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=110602, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=110592, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=110607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=110637, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=110622, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=110627, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=110632, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=110617, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=110597, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([ED0D08669DDD806C]:0)




Build Log:
[...truncated 955 lines...]
   [junit4] Suite: org.apache.lucene.index.TestDuelingCodecsAtNight
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDuelingCodecsAtNight -Dtests.method=testBigEquals -Dtests.seed=FC9E0D956239FF6A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=ar-LB -Dtests.timezone=Africa/Banjul -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1040s J1 | TestDuelingCodecsAtNight.testBigEquals <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FC9E0D956239FF6A:1F9A5610C9223B2D]:0)
   [junit4]    > 	at org.apache.lucene.util.fst.BytesStore.skipBytes(BytesStore.java:307)
   [junit4]    > 	at org.apache.lucene.util.fst.FST.addNode(FST.java:792)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:126)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:214)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:310)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finishTerm(MemoryPostingsFormat.java:254)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.access$500(MemoryPostingsFormat.java:109)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:396)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
   [junit4]    > 	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
   [junit4]    > 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2052)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4928)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:731)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4966)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4957)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1565)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]    > 	at org.apache.lucene.index.TestDuelingCodecs.createRandomIndex(TestDuelingCodecs.java:139)
   [junit4]    > 	at org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals(TestDuelingCodecsAtNight.java:34)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J1/temp/lucene.index.TestDuelingCodecsAtNight_FC9E0D956239FF6A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=494, maxMBSortInHeap=6.178906548772666, sim=RandomSimilarity(queryNorm=false): {titleTokenized=DFR I(n)L1, body=DFR I(ne)B3(800.0)}, locale=ar-LB, timezone=Africa/Banjul
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=225341672,total=468713472
   [junit4]   2> NOTE: All tests run in this JVM: [TestSloppyPhraseQuery2, TestMergedIterator, TestApproximationSearchEquivalence, TestSetOnce, TestSearchWithThreads, TestMergeSchedulerExternal, TestNoMergePolicy, FiniteStringsIteratorTest, TestDocIdSetBuilder, TestVersion, TestSegmentReader, MultiCollectorTest, TestFieldMaskingSpanQuery, TestPerFieldDocValuesFormat, TestIndexSorting, TestSpanTermQuery, TestSortedSetDocValues, TestSearchForDuplicates, TestBlockPostingsFormat3, TestLongBitSet, TestStressAdvance, TestFuzzyQuery, TestOfflineSorter, TestNewestSegment, TestBinaryTerms, TestThreadedForceMerge, TestFieldValueQuery, TestDocValuesRewriteMethod, TestNumericDocValuesUpdates, TestGrowableByteArrayDataOutput, TestMutableValues, TestTermdocPerf, TestBM25Similarity, TestTopDocsMerge, TestTopDocsCollector, TestSpanBoostQuery, TestCharsRef, TestVirtualMethod, TestBoolean2, TestTermVectors, TestStopFilter, TestIndexWriterLockRelease, TestFrequencyTrackingRingBuffer, TestScorerPerf, TestTermVectorsWriter, TestMultiDocValues, TestDuelingCodecsAtNight]
   [junit4] Completed [218/437 (1!)] on J1 in 1065.61s, 3 tests, 1 error <<< FAILURES!

[...truncated 738 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J1-20160817_070821_162.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid15500.hprof ...
   [junit4] Heap dump file created [422554252 bytes in 0.424 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 9428 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/init-core-data-001
   [junit4]   2> 18344 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 18344 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 19150 WARN  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 20380 WARN  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 20612 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 20626 WARN  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 20685 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 20832 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_56757_hdfs____.puu3jy/webapp
   [junit4]   2> 21359 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56757
   [junit4]   2> 22124 WARN  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 22127 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 22131 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_51835_datanode____z4m4x1/webapp
   [junit4]   2> 22637 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51835
   [junit4]   2> 23084 WARN  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 23086 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 23097 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_35106_datanode____fe2hmk/webapp
   [junit4]   2> 23395 INFO  (SUITE-StressHdfsTest-seed#[ED0D08669DDD806C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35106
   [junit4]   2> 25027 INFO  (IPC Server handler 4 on 60850) [    ] BlockStateChange BLOCK* processReport: from storage DS-11537ae8-823b-4538-88c4-b85742c16740 node DatanodeRegistration(127.0.0.1:48839, datanodeUuid=fe9ecf19-8f37-46f5-a816-28a628e40615, infoPort=45741, infoSecurePort=0, ipcPort=56233, storageInfo=lv=-56;cid=testClusterID;nsid=648021315;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs
   [junit4]   2> 25027 INFO  (IPC Server handler 4 on 60850) [    ] BlockStateChange BLOCK* processReport: from storage DS-460a47ab-3fd6-4c2b-bec4-bfb42dead0b5 node DatanodeRegistration(127.0.0.1:48839, datanodeUuid=fe9ecf19-8f37-46f5-a816-28a628e40615, infoPort=45741, infoSecurePort=0, ipcPort=56233, storageInfo=lv=-56;cid=testClusterID;nsid=648021315;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 25040 INFO  (IPC Server handler 0 on 60850) [    ] BlockStateChange BLOCK* processReport: from storage DS-3b777ed2-0744-4dff-bc93-d3f88d4ee8e5 node DatanodeRegistration(127.0.0.1:57880, datanodeUuid=410ac307-983b-4fcf-a1f6-c8eb11e994e3, infoPort=44191, infoSecurePort=0, ipcPort=33287, storageInfo=lv=-56;cid=testClusterID;nsid=648021315;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 25041 INFO  (IPC Server handler 0 on 60850) [    ] BlockStateChange BLOCK* processReport: from storage DS-7c806500-f7a6-413b-b3fe-932da10489cd node DatanodeRegistration(127.0.0.1:57880, datanodeUuid=410ac307-983b-4fcf-a1f6-c8eb11e994e3, infoPort=44191, infoSecurePort=0, ipcPort=33287, storageInfo=lv=-56;cid=testClusterID;nsid=648021315;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 25429 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 25431 INFO  (Thread-167) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 25431 INFO  (Thread-167) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 25530 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.ZkTestServer start zk server on port:59232
   [junit4]   2> 25542 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 25569 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 25611 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2057c2ee name:ZooKeeperConnection Watcher:127.0.0.1:59232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25614 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 25614 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 25615 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 25649 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 25663 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 25671 INFO  (zkCallback-20-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@486b5705 name:ZooKeeperConnection Watcher:127.0.0.1:59232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25675 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 25675 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 25677 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 25682 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 25685 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 25687 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 25693 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 25693 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 25702 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 25702 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 25705 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 25705 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 25709 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 25710 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 25712 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 25712 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 25714 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 25714 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 25718 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 25718 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 25720 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 25720 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 25723 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 25723 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 25730 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 25731 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 25733 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 25733 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 25791 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/cores/collection1
   [junit4]   2> 25793 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 25831 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2d0a3710{/,null,AVAILABLE}
   [junit4]   2> 25832 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@496be49d{HTTP/1.1,[http/1.1]}{127.0.0.1:50360}
   [junit4]   2> 25835 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.e.j.s.Server Started @28033ms
   [junit4]   2> 25835 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:60850/hdfs__localhost_60850__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001_tempDir-002_control_data, hostContext=/, hostPort=50360, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/cores}
   [junit4]   2> 25835 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@5c647e05
   [junit4]   2> 25835 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001'
   [junit4]   2> 25836 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 25836 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 25836 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 25854 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 25866 INFO  (zkCallback-21-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50baef90 name:ZooKeeperConnection Watcher:127.0.0.1:59232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25866 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 25867 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 25877 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 25877 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/solr.xml
   [junit4]   2> 25883 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/cores
   [junit4]   2> 25883 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.CoreContainer New CoreContainer 218496804
   [junit4]   2> 25883 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001]
   [junit4]   2> 25884 WARN  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/lib
   [junit4]   2> 25884 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 25903 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 25903 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 25903 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 25903 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59232/solr
   [junit4]   2> 25930 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 25930 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 25935 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 25956 INFO  (zkCallback-24-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5d77ee39 name:ZooKeeperConnection Watcher:127.0.0.1:59232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25957 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 25957 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 26088 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$82/657884918@7f207340
   [junit4]   2> 26107 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 26125 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4f9e598a name:ZooKeeperConnection Watcher:127.0.0.1:59232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26125 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 26133 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 26139 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 26144 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 26147 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 26150 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 26158 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 26162 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 26166 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 26168 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 26169 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 26173 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 26177 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 26191 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 26197 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 26200 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 26201 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96431487931056132-127.0.0.1:50360_-n_0000000000
   [junit4]   2> 26205 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50360_
   [junit4]   2> 26205 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 26210 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.Overseer Overseer (id=96431487931056132-127.0.0.1:50360_-n_0000000000) starting
   [junit4]   2> 26219 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 26260 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 26261 INFO  (OverseerCollectionConfigSetProcessor-96431487931056132-127.0.0.1:50360_-n_0000000000) [n:127.0.0.1:50360_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 26267 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50360_
   [junit4]   2> 26267 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:50360_
   [junit4]   2> 26267 INFO  (OverseerStateUpdate-96431487931056132-127.0.0.1:50360_-n_0000000000) [n:127.0.0.1:50360_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 26280 INFO  (OverseerStateUpdate-96431487931056132-127.0.0.1:50360_-n_0000000000) [n:127.0.0.1:50360_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 26280 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 26283 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 26449 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 26449 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 26450 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/cores
   [junit4]   2> 26452 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 26452 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/cores/collection1
   [junit4]   2> 26452 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 26468 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0
   [junit4]   2> 26468 INFO  (TEST-StressHdfsTest.test-seed#[ED0D08669DDD806C]) [n:127.0.0.1:50360_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 26469 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 26469 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 26473 INFO  (OverseerStateUpdate-96431487931056132-127.0.0.1:50360_-n_0000000000) [n:127.0.0.1:50360_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:50360",
   [junit4]   2>   "node_name":"127.0.0.1:50360_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 26480 INFO  (OverseerStateUpdate-96431487931056132-127.0.0.1:50360_-n_0000000000) [n:127.0.0.1:50360_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:50360",
   [junit4]   2>   "node_name":"127.0.0.1:50360_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 26483 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 26485 INFO  (OverseerStateUpdate-96431487931056132-127.0.0.1:50360_-n_0000000000) [n:127.0.0.1:50360_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 26488 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 26500 INFO  (OverseerStateUpdate-96431487931056132-127.0.0.1:50360_-n_0000000000) [n:127.0.0.1:50360_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 26509 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 26516 INFO  (zkCallback-25-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 27510 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 27510 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 27511 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 27511 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 27512 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 27513 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/cores/collection1'
   [junit4]   2> 27513 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 27513 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 27515 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 27525 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 27530 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 27553 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 27565 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 27669 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 27684 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 27685 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 27727 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 27730 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 27732 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 27733 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 27735 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0
   [junit4]   2> 27735 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 27735 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 27740 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:60850/solr_hdfs_home
   [junit4]   2> 27740 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 27740 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_ED0D08669DDD806C-001/control-001/cores/collection1], dataDir=[hdfs://localhost:60850/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 27740 INFO  (coreLoadExecutor-110-thread-1-processing-n:127.0.0.1:50360_) [n:127.0.0.1:50360_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ab2375b
   [junit4]   2> 27764 INFO  (coreLo

[...truncated too long message...]

b/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_102/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160817_095505_593.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160817_095505_593.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:984: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 756 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1099 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1099/

4 tests failed.
FAILED:  org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([5EEBCE0191A970B6:B7B175390F30E01E]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:782)
	at org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir(TestArbitraryIndexDir.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=*[count(//doc)=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=id:2&qt=standard&start=0&rows=20&version=2.2
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:775)
	... 41 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 6 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 6 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([5EEBCE0191A970B6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([5EEBCE0191A970B6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=15553, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=15568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=15573, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=15543, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=15558, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=15538, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=15528, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=15533, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=15563, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=15548, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=15553, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=15568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=15573, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=15543, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=15558, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=15538, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=15528, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=15533, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=15563, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=15548, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([5EEBCE0191A970B6]:0)




Build Log:
[...truncated 11211 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/init-core-data-001
   [junit4]   2> 19349 INFO  (SUITE-TestReplicationHandler-seed#[5EEBCE0191A970B6]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 19350 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRateLimitedReplication
   [junit4]   2> 19350 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1
   [junit4]   2> 19363 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 19378 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3c2b9928{/solr,null,AVAILABLE}
   [junit4]   2> 19380 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.ServerConnector Started ServerConnector@2d287ff{HTTP/1.1,[http/1.1]}{127.0.0.1:४२०७१}
   [junit4]   2> 19380 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.Server Started @२१७६४ms
   [junit4]   2> 19380 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=42071}
   [junit4]   2> 19381 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@5c647e05
   [junit4]   2> 19381 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001'
   [junit4]   2> 19381 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19381 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 19381 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/solr.xml
   [junit4]   2> 19394 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/.
   [junit4]   2> 19394 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer New CoreContainer 1821094990
   [junit4]   2> 19394 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001]
   [junit4]   2> 19394 WARN  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/lib
   [junit4]   2> 19396 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 19402 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 19402 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19402 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19402 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 19402 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 19402 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/.
   [junit4]   2> 19403 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 19403 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1
   [junit4]   2> 19403 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 19407 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0
   [junit4]   2> 19407 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 19408 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1'
   [junit4]   2> 19408 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19408 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 19419 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 19421 WARN  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 19422 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 19433 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 19435 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 19441 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 19442 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1'
   [junit4]   2> 19442 INFO  (coreLoadExecutor-160-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1
   [junit4]   2> 19443 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 19444 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 19444 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 19444 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/snapshot_metadata
   [junit4]   2> 19444 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.s.SolrSnapshotMetaDataManager Loading from snapshot metadata file...
   [junit4]   2> 19444 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data
   [junit4]   2> 19444 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index/
   [junit4]   2> 19445 WARN  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 19445 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index
   [junit4]   2> 19445 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@41fd1082
   [junit4]   2> 19446 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7caf0a84 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2cddc427),segFN=segments_1,generation=1}
   [junit4]   2> 19446 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 19449 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 19453 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19454 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19454 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/
   [junit4]   2> 19455 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19456 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19457 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 19457 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,lazy,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file,/terms
   [junit4]   2> 19458 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 19467 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 19467 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 19467 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=4, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.171875, noCFSRatio=0.2995708266391044]
   [junit4]   2> 19468 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7caf0a84 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2cddc427),segFN=segments_1,generation=1}
   [junit4]   2> 19468 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 19468 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5e44f1b[collection1] main]
   [junit4]   2> 19468 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/conf
   [junit4]   2> 19468 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/conf}
   [junit4]   2> 19468 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/conf
   [junit4]   2> 19469 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 19469 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 19469 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 19469 INFO  (coreLoadExecutor-160-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 19470 INFO  (searcherExecutor-161-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5e44f1b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 19909 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1
   [junit4]   2> 19911 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 19915 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3974ccc{/solr,null,AVAILABLE}
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.ServerConnector Started ServerConnector@130f63d{HTTP/1.1,[http/1.1]}{127.0.0.1:३८४९९}
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.Server Started @२२३००ms
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=38499}
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@5c647e05
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002'
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 19916 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/solr.xml
   [junit4]   2> 19922 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/.
   [junit4]   2> 19923 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer New CoreContainer 2035679972
   [junit4]   2> 19923 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002]
   [junit4]   2> 19923 WARN  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/lib
   [junit4]   2> 19923 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 19932 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 19932 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19932 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19932 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 19932 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 19932 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/.
   [junit4]   2> 19933 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 19933 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1
   [junit4]   2> 19933 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 19935 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0
   [junit4]   2> 19935 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 19938 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1'
   [junit4]   2> 19938 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19938 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 19944 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 19946 WARN  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 19947 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 19970 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 19981 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 19983 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 19988 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1'
   [junit4]   2> 19988 INFO  (coreLoadExecutor-170-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1
   [junit4]   2> 19989 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 19989 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 19989 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 19989 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/snapshot_metadata
   [junit4]   2> 19989 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.s.SolrSnapshotMetaDataManager Loading from snapshot metadata file...
   [junit4]   2> 19989 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data
   [junit4]   2> 19990 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/index/
   [junit4]   2> 19990 WARN  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 19990 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 19990 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3cded66d
   [junit4]   2> 19991 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@764085ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c40a4ab),segFN=segments_1,generation=1}
   [junit4]   2> 19991 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 19993 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 19994 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19994 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19995 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19995 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 19997 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 19997 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,lazy,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file,/terms
   [junit4]   2> 19997 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 19997 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 19997 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 19997 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=4, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.171875, noCFSRatio=0.2995708266391044]
   [junit4]   2> 19998 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/
   [junit4]   2> 19998 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@764085ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c40a4ab),segFN=segments_1,generation=1}
   [junit4]   2> 19999 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 19999 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3fef04b9[collection1] main]
   [junit4]   2> 19999 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1/conf
   [junit4]   2> 19999 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1/conf}
   [junit4]   2> 19999 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/collection1/conf
   [junit4]   2> 19999 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 20011 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 20011 INFO  (coreLoadExecutor-170-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 20012 INFO  (searcherExecutor-171-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3fef04b9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 20024 INFO  (qtp1372178669-254) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 20440 INFO  (qtp1372178669-256) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 20441 INFO  (qtp1372178669-256) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 1
   [junit4]   2> 20459 INFO  (qtp1798488711-271) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 20460 INFO  (qtp1798488711-271) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 1
   [junit4]   2> 20462 INFO  (qtp1372178669-257) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 20463 INFO  (qtp1372178669-257) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7caf0a84 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2cddc427),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7caf0a84 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2cddc427),segFN=segments_2,generation=2}
   [junit4]   2> 20463 INFO  (qtp1372178669-257) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 20464 INFO  (qtp1372178669-257) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4beb7c82[collection1] main]
   [junit4]   2> 20464 INFO  (qtp1372178669-257) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 20464 INFO  (searcherExecutor-161-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4beb7c82[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 20464 INFO  (qtp1372178669-257) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 2
   [junit4]   2> 20465 INFO  (qtp1798488711-272) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 20466 INFO  (qtp1798488711-272) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@764085ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c40a4ab),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@764085ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c40a4ab),segFN=segments_2,generation=2}
   [junit4]   2> 20466 INFO  (qtp1798488711-272) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 20467 INFO  (qtp1798488711-272) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2ee410a0[collection1] main]
   [junit4]   2> 20467 INFO  (qtp1798488711-272) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 20467 INFO  (searcherExecutor-171-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2ee410a0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 20467 INFO  (qtp1798488711-272) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 2
   [junit4]   2> 20471 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@2d287ff{HTTP/1.1,[http/1.1]}{127.0.0.1:०}
   [junit4]   2> 20471 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1821094990
   [junit4]   2> 20479 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5fe609d2
   [junit4]   2> 20479 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 3 directories currently being tracked
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index;done=false>>]
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/snapshot_metadata [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/snapshot_metadata;done=false>>]
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 20480 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/snapshot_metadata
   [junit4]   2> 20491 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data;done=false>>]
   [junit4]   2> 20491 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 20491 INFO  (coreCloseExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data
   [junit4]   2> 20492 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3c2b9928{/solr,null,UNAVAILABLE}
   [junit4]   2> 20495 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@130f63d{HTTP/1.1,[http/1.1]}{127.0.0.1:०}
   [junit4]   2> 20495 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2035679972
   [junit4]   2> 20515 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3a34f203
   [junit4]   2> 20516 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 3 directories currently being tracked
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/snapshot_metadata [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/snapshot_metadata;done=false>>]
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 20517 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/snapshot_metadata
   [junit4]   2> 20518 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/index [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/index;done=false>>]
   [junit4]   2> 20518 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 20518 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 20518 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data;done=false>>]
   [junit4]   2> 20518 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 20518 INFO  (coreCloseExecutor-178-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-002/./collection1/data
   [junit4]   2> 20519 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3974ccc{/solr,null,UNAVAILABLE}
   [junit4]   2> 20522 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 20528 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@e988b15{/solr,null,AVAILABLE}
   [junit4]   2> 20529 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6d8b076{HTTP/1.1,[http/1.1]}{127.0.0.1:४१५९१}
   [junit4]   2> 20529 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.e.j.s.Server Started @२२९१३ms
   [junit4]   2> 20529 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=41591}
   [junit4]   2> 20529 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@5c647e05
   [junit4]   2> 20530 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001'
   [junit4]   2> 20530 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 20530 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 20530 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/solr.xml
   [junit4]   2> 20543 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/.
   [junit4]   2> 20543 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer New CoreContainer 1577096232
   [junit4]   2> 20543 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001]
   [junit4]   2> 20543 WARN  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/lib
   [junit4]   2> 20547 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 20559 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 20559 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 20559 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 20559 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 20559 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 20560 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/.
   [junit4]   2> 20560 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 20561 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1
   [junit4]   2> 20561 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 20567 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1'
   [junit4]   2> 20568 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 20568 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 20576 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0
   [junit4]   2> 20576 INFO  (TEST-TestReplicationHandler.testRateLimitedReplication-seed#[5EEBCE0191A970B6]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 20580 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 20583 WARN  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 20583 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 20591 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 20593 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 20596 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 20597 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1'
   [junit4]   2> 20597 INFO  (coreLoadExecutor-185-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1
   [junit4]   2> 20597 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrCore solr.NRTCachingDirectoryFactory
   [junit4]   2> 20600 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 20600 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 20600 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/data/snapshot_metadata
   [junit4]   2> 20600 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.s.SolrSnapshotMetaDataManager Loading from snapshot metadata file...
   [junit4]   2> 20600 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/data
   [junit4]   2> 20600 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index/
   [junit4]   2> 20602 WARN  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 20602 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/data/index
   [junit4]   2> 20605 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7dfdbb04
   [junit4]   2> 20628 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(MMapDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@106e9ccf; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 20628 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 20637 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 20638 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 20638 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 20641 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/./collection1/data/
   [junit4]   2> 20641 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 20642 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 20643 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 20644 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,lazy,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file,/terms
   [junit4]   2> 20644 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 20651 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 20651 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 20652 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@131056e8
   [junit4]   2> 20653 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(MMapDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@106e9ccf; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 20653 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 20653 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2ab1346b[collection1] main]
   [junit4]   2> 20653 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/conf
   [junit4]   2> 20653 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/conf}
   [junit4]   2> 20653 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_5EEBCE0191A970B6-001/solr-instance-001/collection1/conf
   [junit4]   2> 20654 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 20654 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 20654 INFO  (coreLoadExecutor-185-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 20659 INFO  (searcherExecutor-186-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2ab1346b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 21083 INFO  (qtp615894377-293) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 2
   [junit4]   2> 21086 INFO  (qtp615894377-294) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 21088 INFO  (qtp615894377-295) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 21090 INFO  (qtp615894377-296) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 21091 INFO  (qtp615894377-296) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 21093 INFO  (qtp615894377-296) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 21100 INFO  (qtp615894377-293) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 5
   [junit4]   2> 21102 INFO  (qtp615894377-294) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 0
   [junit4]   2> 21104 INFO  (qtp615894377-295) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 21109 INFO  (qtp615894377-297) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 21111 INFO  (qtp615894377-297) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 21112 INFO  (qtp615894377-296) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 21120 INFO  (qtp615894377-296) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 5
   [junit4]   2> 21123 INFO  (qtp615894377-293) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 21125 INFO  (qtp615894377-294) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 21127 INFO  (qtp615894377-294) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  we

[...truncated too long message...]

0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_102/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160816_221444_019.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20160816_221444_019.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:984: At least one slave process threw an exception, first: Forked process returned with error code: 11. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160816_221444_017.sysout See process stderr at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160816_221444_017.syserr

Total time: 330 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1098 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1098/

7 tests failed.
FAILED:  org.apache.solr.cloud.TestLeaderInitiatedRecoveryThread.testPublishDownState

Error Message:
expected:<29> but was:<30>

Stack Trace:
java.lang.AssertionError: expected:<29> but was:<30>
	at __randomizedtesting.SeedInfo.seed([A9C9065E58FF6326:F7B4A4A0FCE22ADB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.TestLeaderInitiatedRecoveryThread.testPublishDownState(TestLeaderInitiatedRecoveryThread.java:114)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10Minutes

Error Message:
Captured an uncaught exception in thread: Thread[id=10053, name=Thread-6582, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10053, name=Thread-6582, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([A9C9065E58FF6326]:0)
	at java.util.HashMap.newNode(HashMap.java:1742)
	at java.util.HashMap.putVal(HashMap.java:630)
	at java.util.HashMap.put(HashMap.java:611)
	at java.util.HashSet.add(HashSet.java:219)
	at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:212)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:243)
	at sun.security.util.AbstractAlgorithmConstraints.checkAlgorithm(AbstractAlgorithmConstraints.java:104)
	at sun.security.util.DisabledAlgorithmConstraints.permits(DisabledAlgorithmConstraints.java:91)
	at sun.security.ssl.SSLAlgorithmConstraints.permits(SSLAlgorithmConstraints.java:144)
	at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:349)
	at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:304)
	at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:626)
	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:567)
	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)


FAILED:  org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([A9C9065E58FF6326:4093BD66C666F38E]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:782)
	at org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir(TestArbitraryIndexDir.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=*[count(//doc)=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">6</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=id:2&qt=standard&start=0&rows=20&version=2.2
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:775)
	... 41 more


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([A9C9065E58FF6326]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=3404, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3404, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:52544: collection already exists: awholynewstresscollection_collection3_2
	at __randomizedtesting.SeedInfo.seed([A9C9065E58FF6326]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:988)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([A9C9065E58FF6326]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=70982, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=70992, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=71002, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=70997, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=71007, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=70972, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=70987, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=71017, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=70977, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=71012, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=70982, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=70992, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=71002, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=70997, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=71007, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=70972, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=70987, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=71017, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=70977, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=71012, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([A9C9065E58FF6326]:0)




Build Log:
[...truncated 11153 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/init-core-data-001
   [junit4]   2> 130319 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 130320 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 130356 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 130361 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 130363 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 130374 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_47960_hdfs____.edghs6/webapp
   [junit4]   2> 130811 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47960
   [junit4]   2> 130871 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 130872 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 130876 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34444_datanode____3uyxwa/webapp
   [junit4]   2> 131200 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34444
   [junit4]   2> 131255 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 131256 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 131264 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_52818_datanode____sulfnt/webapp
   [junit4]   2> 131349 INFO  (IPC Server handler 3 on 57883) [    ] BlockStateChange BLOCK* processReport: from storage DS-abf04536-f518-463e-862b-8e94d836c463 node DatanodeRegistration(127.0.0.1:47919, datanodeUuid=46a87ad8-bafb-468e-895d-0fe0a02a8b60, infoPort=37364, infoSecurePort=0, ipcPort=58064, storageInfo=lv=-56;cid=testClusterID;nsid=1478403124;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 131349 INFO  (IPC Server handler 3 on 57883) [    ] BlockStateChange BLOCK* processReport: from storage DS-734720c8-8578-451c-a9b3-14b8ec0eba7e node DatanodeRegistration(127.0.0.1:47919, datanodeUuid=46a87ad8-bafb-468e-895d-0fe0a02a8b60, infoPort=37364, infoSecurePort=0, ipcPort=58064, storageInfo=lv=-56;cid=testClusterID;nsid=1478403124;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 131556 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A9C9065E58FF6326]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52818
   [junit4]   2> 131645 INFO  (IPC Server handler 8 on 57883) [    ] BlockStateChange BLOCK* processReport: from storage DS-b7a91d45-0f98-4bc0-bd2c-6c2963eee4a6 node DatanodeRegistration(127.0.0.1:51738, datanodeUuid=ad55ad54-f778-4d4c-a2a9-64a4ada42f34, infoPort=47752, infoSecurePort=0, ipcPort=35234, storageInfo=lv=-56;cid=testClusterID;nsid=1478403124;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 131646 INFO  (IPC Server handler 8 on 57883) [    ] BlockStateChange BLOCK* processReport: from storage DS-a8788468-41cd-4f53-95f9-54c7aea3653b node DatanodeRegistration(127.0.0.1:51738, datanodeUuid=ad55ad54-f778-4d4c-a2a9-64a4ada42f34, infoPort=47752, infoSecurePort=0, ipcPort=35234, storageInfo=lv=-56;cid=testClusterID;nsid=1478403124;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 131736 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 131736 INFO  (Thread-1570) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 131736 INFO  (Thread-1570) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 131836 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.ZkTestServer start zk server on port:49874
   [junit4]   2> 131836 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 131837 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131840 INFO  (zkCallback-192-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@126d39ff name:ZooKeeperConnection Watcher:127.0.0.1:49874 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131840 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131840 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 131840 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 131846 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 131848 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131849 INFO  (zkCallback-193-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@68f919f8 name:ZooKeeperConnection Watcher:127.0.0.1:49874/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131849 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131849 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 131849 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 131853 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 131855 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 131856 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 131857 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 131857 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 131859 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 131860 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 131861 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 131861 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 131863 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 131863 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 131864 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 131864 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 131866 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 131866 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 131869 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 131869 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 131875 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 131875 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 131877 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 131877 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 131878 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 131878 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 131880 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 131880 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 131882 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 131883 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131884 INFO  (zkCallback-195-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@122ea86d name:ZooKeeperConnection Watcher:127.0.0.1:49874/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131884 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131884 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 131885 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 131885 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131889 INFO  (zkCallback-196-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@62331959 name:ZooKeeperConnection Watcher:127.0.0.1:49874 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131889 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131889 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 131889 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 131891 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 131891 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131892 INFO  (zkCallback-197-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@22e83304 name:ZooKeeperConnection Watcher:127.0.0.1:49874/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131892 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131893 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 131893 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 131893 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 131895 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 131895 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 131897 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 131897 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 131898 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 131899 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 131900 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 131900 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 131901 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 131902 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 131903 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 131903 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 131905 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 131905 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 131907 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 131907 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 131908 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 131908 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 131910 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 131910 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 131911 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 131911 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 131953 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/cores/collection1
   [junit4]   2> 131956 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 131958 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@55d29294{/,null,AVAILABLE}
   [junit4]   2> 131961 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.e.j.s.ServerConnector Started ServerConnector@45200254{SSL,[ssl, http/1.1]}{127.0.0.1:55025}
   [junit4]   2> 131966 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.e.j.s.Server Started @134007ms
   [junit4]   2> 131967 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:57883/hdfs__localhost_57883__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001_tempDir-002_control_data, hostContext=/, hostPort=55025, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/cores}
   [junit4]   2> 131968 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@5c647e05
   [junit4]   2> 131968 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001'
   [junit4]   2> 131968 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 131968 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 131968 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 131969 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131970 INFO  (zkCallback-198-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4130b5c5 name:ZooKeeperConnection Watcher:127.0.0.1:49874/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131970 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131970 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 131971 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 131971 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/solr.xml
   [junit4]   2> 131975 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/cores
   [junit4]   2> 131976 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.CoreContainer New CoreContainer 976523124
   [junit4]   2> 131976 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001]
   [junit4]   2> 131976 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/lib
   [junit4]   2> 131976 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 131978 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 131978 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 131978 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 131978 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49874/solr
   [junit4]   2> 131978 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 131978 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 131978 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131979 INFO  (zkCallback-201-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5d96e9af name:ZooKeeperConnection Watcher:127.0.0.1:49874 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131980 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131980 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 131981 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$72/1996173683@1f0491d3
   [junit4]   2> 131982 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 131983 INFO  (zkCallback-202-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@76e4f20e name:ZooKeeperConnection Watcher:127.0.0.1:49874/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 131984 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 131984 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 131986 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 131989 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 131992 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 131994 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 131996 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 131997 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 131998 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 131999 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 132000 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 132001 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 132001 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 132004 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 132005 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 132006 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 132006 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96423840087080967-127.0.0.1:55025_-n_0000000000
   [junit4]   2> 132007 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55025_
   [junit4]   2> 132007 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 132008 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.Overseer Overseer (id=96423840087080967-127.0.0.1:55025_-n_0000000000) starting
   [junit4]   2> 132010 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 132015 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 132015 INFO  (OverseerCollectionConfigSetProcessor-96423840087080967-127.0.0.1:55025_-n_0000000000) [n:127.0.0.1:55025_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 132016 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55025_
   [junit4]   2> 132016 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:55025_
   [junit4]   2> 132016 INFO  (OverseerStateUpdate-96423840087080967-127.0.0.1:55025_-n_0000000000) [n:127.0.0.1:55025_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 132018 INFO  (zkCallback-202-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 132018 INFO  (OverseerStateUpdate-96423840087080967-127.0.0.1:55025_-n_0000000000) [n:127.0.0.1:55025_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 132022 INFO  (zkCallback-202-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 132030 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 132030 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 132030 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/cores
   [junit4]   2> 132031 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 132031 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/cores/collection1
   [junit4]   2> 132031 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 132034 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 132034 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 132035 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2
   [junit4]   2> 132035 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[A9C9065E58FF6326]) [n:127.0.0.1:55025_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 132035 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 132036 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 132039 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 132039 INFO  (OverseerStateUpdate-96423840087080967-127.0.0.1:55025_-n_0000000000) [n:127.0.0.1:55025_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:55025",
   [junit4]   2>   "node_name":"127.0.0.1:55025_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 132039 INFO  (OverseerStateUpdate-96423840087080967-127.0.0.1:55025_-n_0000000000) [n:127.0.0.1:55025_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:55025",
   [junit4]   2>   "node_name":"127.0.0.1:55025_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 132039 INFO  (OverseerStateUpdate-96423840087080967-127.0.0.1:55025_-n_0000000000) [n:127.0.0.1:55025_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 132040 INFO  (OverseerStateUpdate-96423840087080967-127.0.0.1:55025_-n_0000000000) [n:127.0.0.1:55025_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 132042 INFO  (zkCallback-202-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 133039 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 133039 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 133040 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 133040 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 133041 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 133041 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/cores/collection1'
   [junit4]   2> 133041 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 133041 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 133042 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 133047 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 133051 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 133061 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 133066 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 133162 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 133169 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 133170 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 133180 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 133181 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 133183 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 133184 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 133185 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2
   [junit4]   2> 133185 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 133185 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 133185 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:57883/solr_hdfs_home
   [junit4]   2> 133185 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 133185 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_A9C9065E58FF6326-001/control-001/cores/collection1], dataDir=[hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 133186 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14c57716
   [junit4]   2> 133186 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 133199 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 133199 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.s.SolrSnapshotMetaDataManager Loading from snapshot metadata file...
   [junit4]   2> 133200 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 133210 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 133211 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 133219 WARN  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:57883/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 133219 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 133231 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 133231 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@26b0e09d
   [junit4]   2> 133273 INFO  (IPC Server handler 5 on 57883) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51738 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-abf04536-f518-463e-862b-8e94d836c463:NORMAL:127.0.0.1:47919|RBW], ReplicaUC[[DISK]DS-a8788468-41cd-4f53-95f9-54c7aea3653b:NORMAL:127.0.0.1:51738|FINALIZED]]} size 0
   [junit4]   2> 133274 INFO  (IPC Server handler 6 on 57883) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:47919 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a8788468-41cd-4f53-95f9-54c7aea3653b:NORMAL:127.0.0.1:51738|FINALIZED], ReplicaUC[[DISK]DS-734720c8-8578-451c-a9b3-14b8ec0eba7e:NORMAL:127.0.0.1:47919|FINALIZED]]} size 0
   [junit4]   2> 133281 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@1acb8c22; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 133281 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 133286 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 133289 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 133289 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 133289 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 133289 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 133289 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 133289 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 133290 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 133290 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 133290 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 133290 WARN  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 133291 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 133292 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 133292 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 133293 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 133297 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/dump,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file,/terms
   [junit4]   2> 133297 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 133301 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 133301 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 133301 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 133312 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 133312 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 133313 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6457253836555219]
   [junit4]   2> 133326 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:57883/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@1acb8c22; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 133326 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 133327 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2c177768[collection1] main]
   [junit4]   2> 133327 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 133328 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 133328 INFO  (coreLoadExecutor-341-thread-1-processing-n:127.0.0.1:55025_) [n:127.0.0.1:55025_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 133328 INFO  

[...truncated too long message...]

nterceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/usr/local/asfpackages/java/jdk1.8.0_102/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160816_012822_651.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160816_012822_651.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Forked process returned with error code: 11. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160816_012822_651.sysout
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1481)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/common-build.xml:984: At least one slave process threw an exception, first: Forked process returned with error code: 11. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J0-20160816_012822_651.sysout

Total time: 264 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1095 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1095/

5 tests failed.
FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
	at __randomizedtesting.SeedInfo.seed([CE72398DF8671AAF:46260657569B7757]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:309)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:127)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([CE72398DF8671AAF:46260657569B7757]:0)
	at java.util.TreeMap.put(TreeMap.java:577)
	at java.util.TreeSet.add(TreeSet.java:255)
	at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:352)
	at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:304)
	at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:626)
	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:567)
	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test(DistributedFacetPivotLongTailTest.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=94031, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=94031, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39655: collection already exists: awholynewstresscollection_collection3_1
	at __randomizedtesting.SeedInfo.seed([CE72398DF8671AAF]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:988)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([CE72398DF8671AAF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=20362, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=20387, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=20367, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=20342, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=20347, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=20372, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=20352, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=20382, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=20357, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=20377, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=20362, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=20387, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=20367, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=20342, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=20347, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=20372, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=20352, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=20382, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=20357, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=20377, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([CE72398DF8671AAF]:0)




Build Log:
[...truncated 11335 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/init-core-data-001
   [junit4]   2> 975998 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 975998 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 976043 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 976055 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 976058 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 976072 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_36854_hdfs____.r83paa/webapp
   [junit4]   2> 976489 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36854
   [junit4]   2> 976563 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 976565 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 976579 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53043_datanode____x7y4zc/webapp
   [junit4]   2> 977001 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53043
   [junit4]   2> 977067 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 977068 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 977083 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53308_datanode____.ov1hmk/webapp
   [junit4]   2> 977145 INFO  (IPC Server handler 3 on 59411) [    ] BlockStateChange BLOCK* processReport: from storage DS-df05d8c6-da49-4e6e-9431-38340585e0b0 node DatanodeRegistration(127.0.0.1:51304, datanodeUuid=d1f91cb0-5a4f-442e-8b15-cf907b5aa424, infoPort=44680, infoSecurePort=0, ipcPort=37162, storageInfo=lv=-56;cid=testClusterID;nsid=1218341561;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 977145 INFO  (IPC Server handler 3 on 59411) [    ] BlockStateChange BLOCK* processReport: from storage DS-4c2c62fb-714d-4ece-b4d6-11ecd1b84e69 node DatanodeRegistration(127.0.0.1:51304, datanodeUuid=d1f91cb0-5a4f-442e-8b15-cf907b5aa424, infoPort=44680, infoSecurePort=0, ipcPort=37162, storageInfo=lv=-56;cid=testClusterID;nsid=1218341561;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 977472 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[CE72398DF8671AAF]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53308
   [junit4]   2> 977567 INFO  (IPC Server handler 2 on 59411) [    ] BlockStateChange BLOCK* processReport: from storage DS-02178ae0-b8f7-4da5-b996-6fc2640cc426 node DatanodeRegistration(127.0.0.1:46154, datanodeUuid=8719dc0d-9246-490a-8e6a-a6b758f5afc8, infoPort=53631, infoSecurePort=0, ipcPort=52363, storageInfo=lv=-56;cid=testClusterID;nsid=1218341561;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 977568 INFO  (IPC Server handler 2 on 59411) [    ] BlockStateChange BLOCK* processReport: from storage DS-146280db-e180-4fa2-b55d-9adbf907b4b6 node DatanodeRegistration(127.0.0.1:46154, datanodeUuid=8719dc0d-9246-490a-8e6a-a6b758f5afc8, infoPort=53631, infoSecurePort=0, ipcPort=52363, storageInfo=lv=-56;cid=testClusterID;nsid=1218341561;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 977640 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 977640 INFO  (Thread-4123) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 977640 INFO  (Thread-4123) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 977740 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.ZkTestServer start zk server on port:38172
   [junit4]   2> 977741 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 977741 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 977744 INFO  (zkCallback-378-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7b8286cf name:ZooKeeperConnection Watcher:127.0.0.1:38172 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 977744 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 977744 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 977744 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 977747 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 977747 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 977749 INFO  (zkCallback-379-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5ca4140d name:ZooKeeperConnection Watcher:127.0.0.1:38172/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 977749 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 977749 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 977749 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 977751 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 977752 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 977753 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 977755 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 977755 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 977757 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 977757 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 977760 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 977760 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 977762 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 977762 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 977764 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 977764 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 977765 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 977765 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 977767 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 977767 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 977769 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 977769 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 977771 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 977771 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 977772 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 977772 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 977774 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 977774 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 977883 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/cores/collection1
   [junit4]   2> 977885 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 977887 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6bafd1ad{/,null,AVAILABLE}
   [junit4]   2> 977888 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.e.j.s.ServerConnector Started ServerConnector@5c214cf4{HTTP/1.1,[http/1.1]}{127.0.0.1:46269}
   [junit4]   2> 977888 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.e.j.s.Server Started @980700ms
   [junit4]   2> 977888 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=46269, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/cores}
   [junit4]   2> 977888 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 977888 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001'
   [junit4]   2> 977889 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 977889 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 977889 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 977889 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 977891 INFO  (zkCallback-380-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2dda127e name:ZooKeeperConnection Watcher:127.0.0.1:38172/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 977891 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 977891 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 977893 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 977893 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/solr.xml
   [junit4]   2> 977899 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/cores
   [junit4]   2> 977899 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.CoreContainer New CoreContainer 1136216840
   [junit4]   2> 977899 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001]
   [junit4]   2> 977899 WARN  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/lib
   [junit4]   2> 977899 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 977902 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 977902 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 977902 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 977902 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38172/solr
   [junit4]   2> 977902 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 977902 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 977902 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 977904 INFO  (zkCallback-383-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6d6a099b name:ZooKeeperConnection Watcher:127.0.0.1:38172 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 977905 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 977905 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 977906 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$9/683162995@753657c0
   [junit4]   2> 977907 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 977908 INFO  (zkCallback-384-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3877bda6 name:ZooKeeperConnection Watcher:127.0.0.1:38172/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 977908 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 977910 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 977915 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 977919 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 977921 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 977922 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 977924 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 977925 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 977926 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 977927 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 977928 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 977929 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 977929 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 977932 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 977933 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 977934 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 977934 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96396525884735492-127.0.0.1:46269_-n_0000000000
   [junit4]   2> 977935 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46269_
   [junit4]   2> 977935 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 977936 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.Overseer Overseer (id=96396525884735492-127.0.0.1:46269_-n_0000000000) starting
   [junit4]   2> 977937 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 977942 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 977943 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 977946 INFO  (OverseerCollectionConfigSetProcessor-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 977947 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46269_
   [junit4]   2> 977947 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:46269_
   [junit4]   2> 977948 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 977948 INFO  (zkCallback-384-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 977949 INFO  (zkCallback-384-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 977964 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 977964 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 977965 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/cores
   [junit4]   2> 977966 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 977966 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/cores/collection1
   [junit4]   2> 977966 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 977967 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 977967 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 977967 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2
   [junit4]   2> 977967 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [n:127.0.0.1:46269_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 977968 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 977968 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46269",
   [junit4]   2>   "node_name":"127.0.0.1:46269_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 977968 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46269",
   [junit4]   2>   "node_name":"127.0.0.1:46269_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 977969 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 977969 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 977969 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 977971 INFO  (zkCallback-384-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 977974 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 977975 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 977975 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 977975 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 977975 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 977976 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 977976 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/cores/collection1'
   [junit4]   2> 977976 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 977976 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 977977 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 977983 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 977987 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 978017 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 978022 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 978099 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 978100 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 978101 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 978103 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 978118 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2
   [junit4]   2> 978118 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 978118 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 978127 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:59411/solr_hdfs_home
   [junit4]   2> 978127 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 978127 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/control-001/cores/collection1], dataDir=[hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 978127 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e3e95dd
   [junit4]   2> 978128 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 978148 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 978148 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 978148 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 980037 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 980037 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 980037 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.s.SolrSnapshotMetaDataManager Loading from snapshot metadata file...
   [junit4]   2> 980039 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 980050 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 980051 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 980061 WARN  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:59411/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 980061 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 980068 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 980068 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 980072 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 980073 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 980073 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1554218371, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 980101 INFO  (IPC Server handler 7 on 59411) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46154 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c2c62fb-714d-4ece-b4d6-11ecd1b84e69:NORMAL:127.0.0.1:51304|RBW], ReplicaUC[[DISK]DS-146280db-e180-4fa2-b55d-9adbf907b4b6:NORMAL:127.0.0.1:46154|RBW]]} size 0
   [junit4]   2> 980102 INFO  (IPC Server handler 3 on 59411) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51304 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-146280db-e180-4fa2-b55d-9adbf907b4b6:NORMAL:127.0.0.1:46154|RBW], ReplicaUC[[DISK]DS-df05d8c6-da49-4e6e-9431-38340585e0b0:NORMAL:127.0.0.1:51304|FINALIZED]]} size 0
   [junit4]   2> 980105 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5eb19cea); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 980105 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 980107 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 980110 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 980110 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 980111 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 980111 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 980111 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 980111 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 980111 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 980111 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 980111 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 980111 WARN  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 980113 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 980114 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 980115 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 980115 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 980120 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/dump,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file,/terms
   [junit4]   2> 980120 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 980125 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 980125 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 980125 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 980137 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 980137 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 980139 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=1.1201171875, floorSegmentMB=0.5703125, forceMergeDeletesPctAllowed=0.8584257569661913, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 980168 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5eb19cea); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 980168 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 980168 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3b69e9e5[collection1] main]
   [junit4]   2> 980168 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 980169 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 980169 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 980169 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 980169 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 980169 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 980170 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 980170 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 980170 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 980172 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 980174 INFO  (searcherExecutor-980-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3b69e9e5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 980174 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 980174 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 980174 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 980174 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1542344416810762240
   [junit4]   2> 980176 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1542344416810762240
   [junit4]   2> 980176 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 980176 INFO  (coreLoadExecutor-979-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 980177 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:46269 collection:control_collection shard:shard1
   [junit4]   2> 980177 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 980180 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/96396525884735492-core_node1-n_0000000000
   [junit4]   2> 980180 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 980182 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 980182 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 980182 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46269/collection1/
   [junit4]   2> 980182 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 980182 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46269/collection1/ has no replicas
   [junit4]   2> 980182 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 980183 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 980183 INFO  (zkCallback-384-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 980185 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/96396525884735492-core_node1-n_0000000000
   [junit4]   2> 980187 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46269/collection1/ shard1
   [junit4]   2> 980187 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:46269",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 980290 INFO  (zkCallback-384-thread-1-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 980338 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:46269/collection1/ and leader is http://127.0.0.1:46269/collection1/
   [junit4]   2> 980338 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46269
   [junit4]   2> 980338 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 980338 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 980338 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 980339 INFO  (coreZkRegister-973-thread-1-processing-n:127.0.0.1:46269_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:46269_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Checking legacy cluster state for collection control_collection
   [junit4]   2> 980340 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "dataDir":"hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46269",
   [junit4]   2>   "node_name":"127.0.0.1:46269_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "ulogDir":"hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/tlog"} current state version: 3
   [junit4]   2> 980340 INFO  (OverseerStateUpdate-96396525884735492-127.0.0.1:46269_-n_0000000000) [n:127.0.0.1:46269_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "dataDir":"hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46269",
   [junit4]   2>   "node_name":"127.0.0.1:46269_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "ulogDir":"hdfs://localhost:59411/solr_hdfs_home/control_collection/core_node1/data/tlog"}
   [junit4]   2> 980396 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 980397 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 980400 INFO  (zkCallback-388-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ca5ca1c name:ZooKeeperConnection Watcher:127.0.0.1:38172/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 980400 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 980400 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 980400 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 980402 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 980403 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 980404 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 980442 INFO  (zkCallback-384-thread-2-processing-n:127.0.0.1:46269_) [n:127.0.0.1:46269_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 980442 INFO  (zkCallback-388-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 980494 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/shard-1-001/cores/collection1
   [junit4]   2> 980495 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_CE72398DF8671AAF-001/shard-1-001
   [junit4]   2> 980495 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 980497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31ea10fb{/,null,AVAILABLE}
   [junit4]   2> 980497 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[CE72398DF8671AAF]) [    ] o.e.j.s.

[...truncated too long message...]

ntextHandler@57e6867a{/_/n,null,UNAVAILABLE}
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedFacetPivotLongTailTest -Dtests.method=test -Dtests.seed=CE72398DF8671AAF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=th-TH -Dtests.timezone=Etc/GMT+0 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   28.8s J0 | DistributedFacetPivotLongTailTest.test <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CE72398DF8671AAF:46260657569B7757]:0)
   [junit4]    > 	at java.util.TreeMap.put(TreeMap.java:577)
   [junit4]    > 	at java.util.TreeSet.add(TreeSet.java:255)
   [junit4]    > 	at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:352)
   [junit4]    > 	at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:304)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:626)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:567)
   [junit4]    > 	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
   [junit4]    > 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
   [junit4]    > 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
   [junit4]    > 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
   [junit4]    > 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
   [junit4]    > 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]    > 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]    > 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]    > 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
   [junit4]    > 	at org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test(DistributedFacetPivotLongTailTest.java:71)
   [junit4]   2> 4560846 INFO  (SUITE-DistributedFacetPivotLongTailTest-seed#[CE72398DF8671AAF]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotLongTailTest_CE72398DF8671AAF-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=ClassicSimilarity, locale=th-TH, timezone=Etc/GMT+0
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=119917392,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [ParsingFieldUpdateProcessorsTest, QueryElevationComponentTest, TestBulkSchemaConcurrent, TestMergePolicyConfig, DistributedSpellCheckComponentTest, SynonymTokenizerTest, TestJoin, TestSizeLimitedDistributedMap, TestRandomDVFaceting, TestStressLucene, AddSchemaFieldsUpdateProcessorFactoryTest, TestReRankQParserPlugin, BooleanFieldTest, WrapperMergePolicyFactoryTest, TestSolrCloudWithSecureImpersonation, SOLR749Test, TestBlobHandler, TestConfigSetImmutable, AsyncCallRequestStatusResponseTest, TestRangeQuery, ActionThrottleTest, JSONWriterTest, CacheHeaderTest, DistributedVersionInfoTest, HdfsThreadLeakTest, TestCSVLoader, UUIDFieldTest, BasicDistributedZkTest, DistributedSuggestComponentTest, StandardRequestHandlerTest, CopyFieldTest, OverseerRolesTest, RecoveryAfterSoftCommitTest, SolrCloudExampleTest, TestInitParams, TestBulkSchemaAPI, TestPerFieldSimilarityWithDefaultOverride, ConnectionManagerTest, TestFreeTextSuggestions, OverriddenZkACLAndCredentialsProvidersTest, TriLevelCompositeIdRoutingTest, TestFieldCacheSortRandom, TestFieldCacheReopen, TestSchemaSimilarityResource, TestBadConfig, TestAnalyzedSuggestions, DeleteShardTest, TestAnalyzeInfixSuggestions, PreAnalyzedFieldTest, TestDocSet, RAMDirectoryFactoryTest, TestSha256AuthenticationProvider, SmileWriterTest, SharedFSAutoReplicaFailoverUtilsTest, TestAuthorizationFramework, TestRequestStatusCollectionAPI, CursorMarkTest, HdfsBasicDistributedZkTest, TestCollapseQParserPlugin, DeleteInactiveReplicaTest, ResourceLoaderTest, TestUninvertingReader, JsonLoaderTest, HLLSerializationTest, DisMaxRequestHandlerTest, SolrRequestParserTest, RestartWhileUpdatingTest, QueryResultKeyTest, DocValuesMissingTest, DefaultValueUpdateProcessorTest, TestCloudSchemaless, DirectoryFactoryTest, CdcrRequestHandlerTest, TestOrdValues, TestDistribIDF, SignatureUpdateProcessorFactoryTest, TestLazyCores, HdfsCollectionsAPIDistributedZkTest, TestReplicationHandler, ZkStateReaderTest, DistributedFacetPivotWhiteBoxTest, TestCloudPseudoReturnFields, TestBinaryField, TestLRUCache, IndexSchemaRuntimeFieldTest, DistribDocExpirationUpdateProcessorTest, TestSubQueryTransformer, TestConfig, TestLMDirichletSimilarityFactory, TestShardHandlerFactory, TolerantUpdateProcessorTest, TestSolrQueryResponse, SortSpecParsingTest, ZkNodePropsTest, LeaderElectionIntegrationTest, SuggestComponentContextFilterQueryTest, DistributedQueryElevationComponentTest, PolyFieldTest, TestCloudDeleteByQuery, TestConfigSetsAPIExclusivity, HdfsLockFactoryTest, SaslZkACLProviderTest, BlockJoinFacetDistribTest, TestLocalFSCloudBackupRestore, TestDistributedGrouping, TestReload, TestSolr4Spatial, TestGroupingSearch, TestFunctionQuery, BasicFunctionalityTest, SoftAutoCommitTest, TestFoldingMultitermQuery, NoCacheHeaderTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, QueryEqualityTest, TestRemoteStreaming, XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, RequestHandlersTest, TestValueSourceCache, TestIndexingPerformance, TestArbitraryIndexDir, SolrPluginUtilsTest, TestJmxIntegration, TestCSVResponseWriter, TestPartialUpdateDeduplication, PingRequestHandlerTest, TestBinaryResponseWriter, TestLFUCache, CursorPagingTest, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, AliasIntegrationTest, CdcrReplicationHandlerTest, CollectionsAPISolrJTest, DeleteReplicaTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnCommitTest, MigrateRouteKeyTest, ReplicationFactorTest, SimpleCollectionCreateDeleteTest, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestDownShardTolerantSearch, TestLeaderElectionZkExpiry, TestLockTree, TestMiniSolrCloudClusterBase, TestMiniSolrCloudClusterSSL, TestRebalanceLeaders, TestSolrCloudWithDelegationTokens, TestSolrCloudWithKerberosAlt, TestTolerantUpdateProcessorCloud, HdfsBasicDistributedZk2Test, HdfsChaosMonkeyNothingIsSafeTest, HdfsRestartWhileUpdatingTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest, TestClusterStateMutator, RuleEngineTest, BlobRepositoryCloudTest, HdfsDirectoryFactoryTest, TestBackupRepositoryFactory, TestDynamicLoading, TestNRTOpen, TestSolrXml, TestSolrCoreSnapshots, TestConfigReload, TestHdfsBackupRestoreCore, TestReplicationHandlerBackup, TestRestoreCore, TestSQLHandler, CoreAdminCreateDiscoverTest, InfoHandlerTest, SecurityConfHandlerTest, DistributedDebugComponentTest, DistributedExpandComponentTest, DistributedFacetPivotLargeTest, DistributedFacetPivotLongTailTest]
   [junit4] Completed [509/627 (4!)] on J0 in 28.85s, 1 test, 1 error <<< FAILURES!

[...truncated 358 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J0-20160811_052754_321.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps/java_pid26861.hprof ...
   [junit4] Heap dump file created [495108662 bytes in 1.706 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 11043 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:715: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid26861.hprof

Total time: 263 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1094 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1094/

11 tests failed.
FAILED:  org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterRestart

Error Message:
Timeout waiting for CDCR replication to complete @source_collection:shard1

Stack Trace:
java.lang.RuntimeException: Timeout waiting for CDCR replication to complete @source_collection:shard1
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E:4227993EB804AF78]:0)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForReplicationToComplete(BaseCdcrDistributedZkTest.java:794)
	at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testReplicationAfterRestart(CdcrReplicationDistributedZkTest.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RestartWhileUpdatingTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=155522, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-RestartWhileUpdatingTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=155522, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-RestartWhileUpdatingTest]
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E:966E4165D8D876B6]:0)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.TestHdfsCloudBackupRestore.test

Error Message:
Error from server at http://127.0.0.1:42255/solr: Collection 'hdfsbackuprestore_restored' exists, no action taken.

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:42255/solr: Collection 'hdfsbackuprestore_restored' exists, no action taken.
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E:966E4165D8D876B6]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:232)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletesViaShard1NonLeaderClient

Error Message:
failed opportunistic concurrent delete by id: exists: Num error ids: [] expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: failed opportunistic concurrent delete by id: exists: Num error ids: [] expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E:C5EA69A4DED8F1E9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertUpdateTolerantErrors(TestTolerantUpdateProcessorCloud.java:974)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletes(TestTolerantUpdateProcessorCloud.java:367)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletesViaShard1NonLeaderClient(TestTolerantUpdateProcessorCloud.java:294)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testAddsMixedWithDeletesViaShard1LeaderClient

Error Message:
IOException occured when talking to server at: https://127.0.0.1:60060/solr/test_col

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:60060/solr/test_col
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E:94BACC3D248EB696]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.clearCollection(TestTolerantUpdateProcessorCloud.java:225)
	at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:905)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 49 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at sun.security.ssl.InputRecord.read(InputRecord.java:505)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	... 65 more


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testSanity

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:48335

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:48335
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 47 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=2740, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsRestartWhileUpdatingTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2740, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsRestartWhileUpdatingTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.security.BasicAuthIntegrationTest.testBasics

Error Message:
IOException occured when talking to server at: http://127.0.0.1:49611/solr/testSolrCloudCollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException occured when talking to server at: http://127.0.0.1:49611/solr/testSolrCloudCollection_shard1_replica1
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E:23E2D0934ECA453E]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:760)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.security.BasicAuthIntegrationTest.doExtraTests(BasicAuthIntegrationTest.java:193)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testCollectionCreateSearchDelete(TestMiniSolrCloudClusterBase.java:196)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testBasics(TestMiniSolrCloudClusterBase.java:79)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:49611/solr/testSolrCloudCollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:735)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:49611 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 10 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=468, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=483, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=458, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=478, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=453, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=463, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=488, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=493, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=473, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=448, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=468, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=483, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=458, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=478, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=453, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=463, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=488, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=493, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=473, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=448, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([1E3A7EBF76241B4E]:0)




Build Log:
[...truncated 11163 lines...]
   [junit4] Suite: org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/init-core-data-001
   [junit4]   2> 61931 INFO  (SUITE-ConcurrentDeleteAndCreateCollectionTest-seed#[1E3A7EBF76241B4E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 61934 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testConcurrentCreateAndDeleteOverTheSameConfig
   [junit4]   2> 61936 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 61942 INFO  (Thread-47) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 61942 INFO  (Thread-47) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 62042 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.ZkTestServer start zk server on port:37065
   [junit4]   2> 62042 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62058 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62086 INFO  (zkCallback-46-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@51fbfd4d name:ZooKeeperConnection Watcher:127.0.0.1:37065 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62086 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62086 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62086 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 62383 INFO  (jetty-launcher-45-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 62441 INFO  (jetty-launcher-45-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@735189be{/solr,null,AVAILABLE}
   [junit4]   2> 62458 INFO  (jetty-launcher-45-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@69c12b07{HTTP/1.1,[http/1.1]}{127.0.0.1:35483}
   [junit4]   2> 62458 INFO  (jetty-launcher-45-thread-1) [    ] o.e.j.s.Server Started @64835ms
   [junit4]   2> 62458 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35483}
   [junit4]   2> 62472 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 62473 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1'
   [junit4]   2> 62473 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 62473 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 62473 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62490 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62510 INFO  (zkCallback-47-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e6a727a name:ZooKeeperConnection Watcher:127.0.0.1:37065/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62510 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62510 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62511 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 62540 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/.
   [junit4]   2> 62540 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 72878396
   [junit4]   2> 62540 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1]
   [junit4]   2> 62540 WARN  (jetty-launcher-45-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/lib
   [junit4]   2> 62560 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 62573 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 62579 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 62580 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 62580 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37065/solr
   [junit4]   2> 62590 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 62591 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62591 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62710 INFO  (zkCallback-50-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@22ba3969 name:ZooKeeperConnection Watcher:127.0.0.1:37065 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62710 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62710 INFO  (jetty-launcher-45-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62713 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$87/33072513@402b9b2b
   [junit4]   2> 62723 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62730 INFO  (zkCallback-51-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@32b83625 name:ZooKeeperConnection Watcher:127.0.0.1:37065/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62730 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62737 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 62744 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 62747 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 62749 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 62756 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 62764 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 62772 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 62784 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 62786 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 62788 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 62789 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 62790 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 62791 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 62807 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 62808 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 62811 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 62812 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96391951913254915-127.0.0.1:35483_solr-n_0000000000
   [junit4]   2> 62812 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35483_solr
   [junit4]   2> 62813 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 62814 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.Overseer Overseer (id=96391951913254915-127.0.0.1:35483_solr-n_0000000000) starting
   [junit4]   2> 62819 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 62846 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 62849 INFO  (OverseerCollectionConfigSetProcessor-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 62850 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35483_solr
   [junit4]   2> 62850 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:35483_solr
   [junit4]   2> 62851 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 62851 INFO  (zkCallback-51-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 62853 INFO  (zkCallback-51-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 62853 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 62937 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 62937 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 62942 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/.
   [junit4]   2> 62943 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 62957 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1
   [junit4]   2> 62957 INFO  (jetty-launcher-45-thread-1) [n:127.0.0.1:35483_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 62957 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62962 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62982 INFO  (zkCallback-53-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6a7a2f83 name:ZooKeeperConnection Watcher:127.0.0.1:37065 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62982 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62982 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62999 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 63003 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 63006 INFO  (zkCallback-55-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@677e9a7a name:ZooKeeperConnection Watcher:127.0.0.1:37065/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 63006 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 63006 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 63012 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/testconfig/schema.xml
   [junit4]   2> 63017 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[1E3A7EBF76241B4E]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/testconfig/solrconfig.xml
   [junit4]   2> 63394 INFO  (qtp728730816-319) [n:127.0.0.1:35483_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=testconfig&name=collection0&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 63394 INFO  (qtp728730816-320) [n:127.0.0.1:35483_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=testconfig&name=collection1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 63412 INFO  (OverseerCollectionConfigSetProcessor-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>   "name":"collection0",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"testconfig",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 63420 INFO  (OverseerCollectionConfigSetProcessor-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000001 message:{
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"testconfig",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 63421 INFO  (OverseerThreadFactory-155-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection0",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"testconfig",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 63422 INFO  (OverseerThreadFactory-155-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler Create collection collection0
   [junit4]   2> 63437 INFO  (OverseerThreadFactory-155-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/collection0 
   [junit4]   2> 63438 INFO  (OverseerThreadFactory-155-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection0
   [junit4]   2> 63438 INFO  (OverseerThreadFactory-155-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"testconfig",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 63438 INFO  (OverseerThreadFactory-155-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler Create collection collection1
   [junit4]   2> 63442 INFO  (OverseerThreadFactory-155-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/collection1 
   [junit4]   2> 63443 INFO  (OverseerThreadFactory-155-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 63443 INFO  (zkCallback-51-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 63444 INFO  (zkCallback-51-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 63454 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"collection0",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"testconfig",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 63455 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection0
   [junit4]   2> 63469 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection0/state.json
   [junit4]   2> 63473 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"testconfig",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 63473 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 63548 INFO  (OverseerThreadFactory-155-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating SolrCores for new collection collection0, shardNames [shard1] , replicationFactor : 1
   [junit4]   2> 63548 INFO  (OverseerThreadFactory-155-thread-1-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating core collection0_shard1_replica1 as part of shard shard1 of collection collection0 on 127.0.0.1:35483_solr
   [junit4]   2> 63569 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=testconfig&name=collection0_shard1_replica1&action=CREATE&numShards=1&collection=collection0&shard=shard1&wt=javabin&version=2
   [junit4]   2> 63571 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection0_shard1_replica1, collection.configName=testconfig, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection0, numShards=1, dataDir=data/, shard=shard1}
   [junit4]   2> 63749 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 63749 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 63753 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.ZkController Registering watch for collection collection0
   [junit4]   2> 63758 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.c.ZkStateReader Add data for [collection0] ver [0]
   [junit4]   2> 63759 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 63760 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection0_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35483/solr",
   [junit4]   2>   "node_name":"127.0.0.1:35483_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection0",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 63762 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection0_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35483/solr",
   [junit4]   2>   "node_name":"127.0.0.1:35483_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection0",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 63850 INFO  (OverseerThreadFactory-155-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating SolrCores for new collection collection1, shardNames [shard1] , replicationFactor : 1
   [junit4]   2> 63850 INFO  (OverseerThreadFactory-155-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating core collection1_shard1_replica1 as part of shard shard1 of collection collection1 on 127.0.0.1:35483_solr
   [junit4]   2> 63864 INFO  (qtp728730816-317) [n:127.0.0.1:35483_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=testconfig&name=collection1_shard1_replica1&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2
   [junit4]   2> 63865 INFO  (qtp728730816-317) [n:127.0.0.1:35483_solr    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1_shard1_replica1, collection.configName=testconfig, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, numShards=1, dataDir=data/, shard=shard1}
   [junit4]   2> 63865 INFO  (qtp728730816-317) [n:127.0.0.1:35483_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 63866 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection0/state.json version: 0
   [junit4]   2> 63867 INFO  (qtp728730816-317) [n:127.0.0.1:35483_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 63867 INFO  (zkCallback-51-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection0/state.json] for collection [collection0] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 63870 INFO  (qtp728730816-317) [n:127.0.0.1:35483_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.c.ZkStateReader Add data for [collection1] ver [0]
   [junit4]   2> 63870 INFO  (qtp728730816-317) [n:127.0.0.1:35483_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 63870 INFO  (zkCallback-51-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader Updating data for [collection0] from [0] to [1]
   [junit4]   2> 63871 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35483/solr",
   [junit4]   2>   "node_name":"127.0.0.1:35483_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 63872 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35483/solr",
   [junit4]   2>   "node_name":"127.0.0.1:35483_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 63975 INFO  (OverseerStateUpdate-96391951913254915-127.0.0.1:35483_solr-n_0000000000) [n:127.0.0.1:35483_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 63976 INFO  (zkCallback-51-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 63976 INFO  (zkCallback-51-thread-2-processing-n:127.0.0.1:35483_solr) [n:127.0.0.1:35483_solr    ] o.a.s.c.c.ZkStateReader Updating data for [collection1] from [0] to [1]
   [junit4]   2> 64759 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection0_shard1_replica1
   [junit4]   2> 64759 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.ZkController Check for collection zkNode:collection0
   [junit4]   2> 64760 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 64760 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/collection0]
   [junit4]   2> 64761 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.c.ZkStateReader path=[/collections/collection0] [configName]=[testconfig] specified config exists in ZooKeeper
   [junit4]   2> 64762 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/collection0_shard1_replica1'
   [junit4]   2> 64762 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 64762 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 64765 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 64769 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 64790 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 64794 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 64797 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.s.IndexSchema [collection0_shard1_replica1] Schema name=minimal
   [junit4]   2> 64799 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 64800 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/testconfig', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1
   [junit4]   2> 64800 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1  x:collection0_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'collection0_shard1_replica1' using configuration from collection collection0
   [junit4]   2> 64801 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 64801 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrCore [[collection0_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/collection0_shard1_replica1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data/]
   [junit4]   2> 64801 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection0_shard1_replica1
   [junit4]   2> 64802 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data/snapshot_metadata
   [junit4]   2> 64802 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.s.SolrSnapshotMetaDataManager Loading from snapshot metadata file...
   [junit4]   2> 64802 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data
   [junit4]   2> 64802 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data/index/
   [junit4]   2> 64802 WARN  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrCore [collection0_shard1_replica1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 64802 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data/index
   [junit4]   2> 64803 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@5777e6dd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@506bb336),segFN=segments_1,generation=1}
   [junit4]   2> 64803 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 64804 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 64806 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 64807 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 64807 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 64808 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 64808 INFO  (OldIndexDirectoryCleanupThreadForCore-collection0_shard1_replica1) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection0_shard1_replica1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data/
   [junit4]   2> 64808 WARN  (OldIndexDirectoryCleanupThreadForCore-collection0_shard1_replica1) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_1E3A7EBF76241B4E-001/tempDir-001/node1/./collection0_shard1_replica1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 64809 INFO  (qtp728730816-323) [n:127.0.0.1:35483_solr c:collection0 s:shard1 r:core_node1 x:collection0_shard1_replica1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/update/csv,/sql,/graph,/select,/admin/segments,/admin/system,/replication,/config,/stream,/schema,/admin/plugins,/admin/logging,

[...truncated too long message...]

lr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160810_101957_203.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160810_101957_203.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 235 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
ERROR: Failed to archive {README.txt=README.txt, solr/build/solr-core/test/temp/junit4-J2-20160810_101957_203.events=solr/build/solr-core/test/temp/junit4-J2-20160810_101957_203.events, heapdumps/java_pid30309.hprof=heapdumps/java_pid30309.hprof, heapdumps/java_pid30312.hprof=heapdumps/java_pid30312.hprof, heapdumps/java_pid30311.hprof=heapdumps/java_pid30311.hprof, solr/build/solr-core/test/temp/junit4-J1-20160810_101957_203.events=solr/build/solr-core/test/temp/junit4-J1-20160810_101957_203.events, solr/build/solr-core/test/temp/junit4-J0-20160810_101957_203.events=solr/build/solr-core/test/temp/junit4-J0-20160810_101957_203.events} due to internal error; falling back to full archiving
java.lang.NullPointerException
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1093 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1093/

4 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsSyncSliceTest.test

Error Message:
Error from server at https://127.0.0.1:55852/kf_z/gw/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /kf_z/gw/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=java.lang.OutOfMemoryError},msg=SolrCore &apos;collection1&apos; is not available due to init failure: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1114)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:414)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:815)  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:688)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:860)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:516)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more Caused by: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.  at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:304)  at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:286)  at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:220)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:426)  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:756)  ... 8 more Caused by: java.lang.OutOfMemoryError: Direct buffer memory  at java.nio.Bits.reserveMemory(Bits.java:693)  at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)  at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)  at org.apache.solr.store.blockcache.BlockCache.&lt;init&gt;(BlockCache.java:63)  at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:302)  ... 13 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:55852/kf_z/gw/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /kf_z/gw/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=java.lang.OutOfMemoryError},msg=SolrCore &apos;collection1&apos; is not available due to init failure: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1114)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:414)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:815)
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:688)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:860)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:516)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=&lt;size&gt;g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:304)
	at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:286)
	at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:220)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346)
	at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:426)
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:756)
	... 8 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:693)
	at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at org.apache.solr.store.blockcache.BlockCache.&lt;init&gt;(BlockCache.java:63)
	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:302)
	... 13 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([F83E282DF373B084:706A17F75D8FDD7C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:576)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.SyncSliceTest.test(SyncSliceTest.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:45156: Error CREATEing SolrCore 'test_unload_shard_and_collection_2': Unable to create core [test_unload_shard_and_collection_2] Caused by: Direct buffer memory

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45156: Error CREATEing SolrCore 'test_unload_shard_and_collection_2': Unable to create core [test_unload_shard_and_collection_2] Caused by: Direct buffer memory
	at __randomizedtesting.SeedInfo.seed([F83E282DF373B084:706A17F75D8FDD7C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:137)
	at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([F83E282DF373B084]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=3615, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=3630, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=3610, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=3635, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=3650, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=3625, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=3645, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=3605, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=3620, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=3640, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=3615, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=3630, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=3610, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=3635, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=3650, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=3625, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=3645, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=3605, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=3620, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=3640, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([F83E282DF373B084]:0)




Build Log:
[...truncated 11158 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsSyncSliceTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/init-core-data-001
   [junit4]   2> 457019 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 457022 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /kf_z/gw
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 457059 WARN  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 457067 WARN  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 457069 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 457079 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_45425_hdfs____.6iflkw/webapp
   [junit4]   2> 457458 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45425
   [junit4]   2> 457525 WARN  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 457526 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 457535 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_38509_datanode____4797mo/webapp
   [junit4]   2> 457880 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38509
   [junit4]   2> 457921 WARN  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 457925 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 457940 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_45213_datanode____.24pbdy/webapp
   [junit4]   2> 457997 INFO  (IPC Server handler 4 on 45781) [    ] BlockStateChange BLOCK* processReport: from storage DS-50075429-e057-4a8f-a870-9ce0c83216b1 node DatanodeRegistration(127.0.0.1:43806, datanodeUuid=541b819c-b317-4bcd-a66d-bc36f13e7277, infoPort=46894, infoSecurePort=0, ipcPort=36152, storageInfo=lv=-56;cid=testClusterID;nsid=1513994612;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 457997 INFO  (IPC Server handler 4 on 45781) [    ] BlockStateChange BLOCK* processReport: from storage DS-9d62c35d-f243-4c2c-ad8a-9aedf8097be8 node DatanodeRegistration(127.0.0.1:43806, datanodeUuid=541b819c-b317-4bcd-a66d-bc36f13e7277, infoPort=46894, infoSecurePort=0, ipcPort=36152, storageInfo=lv=-56;cid=testClusterID;nsid=1513994612;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 458295 INFO  (SUITE-HdfsSyncSliceTest-seed#[F83E282DF373B084]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45213
   [junit4]   2> 458384 INFO  (IPC Server handler 0 on 45781) [    ] BlockStateChange BLOCK* processReport: from storage DS-fe1b1872-2093-48c6-a72e-c1adaa32b973 node DatanodeRegistration(127.0.0.1:59488, datanodeUuid=b4c490c8-013a-4c04-bf9e-4b8da40183a3, infoPort=55663, infoSecurePort=0, ipcPort=45766, storageInfo=lv=-56;cid=testClusterID;nsid=1513994612;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 458385 INFO  (IPC Server handler 0 on 45781) [    ] BlockStateChange BLOCK* processReport: from storage DS-7c879716-85ed-483c-a5f2-5d204336af32 node DatanodeRegistration(127.0.0.1:59488, datanodeUuid=b4c490c8-013a-4c04-bf9e-4b8da40183a3, infoPort=55663, infoSecurePort=0, ipcPort=45766, storageInfo=lv=-56;cid=testClusterID;nsid=1513994612;c=0), blocks: 0, hasStaleStorage: false, processing time: 1 msecs
   [junit4]   2> 458454 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 458455 INFO  (Thread-1304) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 458455 INFO  (Thread-1304) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 458555 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.ZkTestServer start zk server on port:55825
   [junit4]   2> 458555 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 458555 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 458558 INFO  (zkCallback-1342-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@874f97f name:ZooKeeperConnection Watcher:127.0.0.1:55825 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 458558 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 458558 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 458558 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 458565 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 458573 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 458574 INFO  (zkCallback-1343-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e82aebb name:ZooKeeperConnection Watcher:127.0.0.1:55825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 458574 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 458575 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 458575 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 458576 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 458577 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 458585 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 458585 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 458585 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 458586 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 458586 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 458587 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 458587 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 458588 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 458588 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 458589 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 458589 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 458590 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 458590 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 458591 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 458591 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 458592 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 458592 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 458593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 458593 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 458600 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 458600 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 458602 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 458602 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 458603 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 458604 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 458605 INFO  (zkCallback-1345-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@339f48fe name:ZooKeeperConnection Watcher:127.0.0.1:55825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 458605 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 458606 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 458667 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/cores/collection1
   [junit4]   2> 458668 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 458669 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31b827b1{/kf_z/gw,null,AVAILABLE}
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.ServerConnector Started ServerConnector@11ad6fb7{SSL,[ssl, http/1.1]}{127.0.0.1:55852}
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.Server Started @461075ms
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:45781/hdfs__localhost_45781__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001_tempDir-002_control_data, hostContext=/kf_z/gw, hostPort=55852, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/cores}
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001'
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 458672 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 458674 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 458681 INFO  (zkCallback-1346-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@245ca77a name:ZooKeeperConnection Watcher:127.0.0.1:55825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 458681 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 458681 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 458682 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 458682 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/solr.xml
   [junit4]   2> 458701 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/cores
   [junit4]   2> 458701 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.CoreContainer New CoreContainer 1287925369
   [junit4]   2> 458701 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001]
   [junit4]   2> 458701 WARN  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/lib
   [junit4]   2> 458701 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 458702 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 458703 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 458703 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 458703 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55825/solr
   [junit4]   2> 458703 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 458703 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 458703 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 458704 INFO  (zkCallback-1349-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@d57adbb name:ZooKeeperConnection Watcher:127.0.0.1:55825 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 458704 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 458704 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 458705 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$70/1016693972@34e1c19c
   [junit4]   2> 458706 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 458706 INFO  (zkCallback-1350-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@774deabf name:ZooKeeperConnection Watcher:127.0.0.1:55825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 458706 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 458707 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 458708 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 458709 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 458710 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 458711 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 458712 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 458713 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 458713 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 458714 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 458714 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 458714 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 458715 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 458716 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 458717 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 458717 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 458717 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000
   [junit4]   2> 458718 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55852_kf_z%2Fgw
   [junit4]   2> 458718 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 458718 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.Overseer Overseer (id=96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) starting
   [junit4]   2> 458719 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 458722 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 458723 INFO  (OverseerStateUpdate-96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 458723 INFO  (OverseerCollectionConfigSetProcessor-96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 458725 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55852_kf_z%2Fgw
   [junit4]   2> 458725 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:55852_kf_z%2Fgw
   [junit4]   2> 458725 INFO  (OverseerStateUpdate-96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 458727 INFO  (zkCallback-1350-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 458728 INFO  (zkCallback-1350-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 458735 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 458735 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 458735 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/cores
   [junit4]   2> 458736 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 458736 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/cores/collection1
   [junit4]   2> 458736 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 458741 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 458741 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 458741 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2
   [junit4]   2> 458741 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 458749 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 458749 INFO  (OverseerStateUpdate-96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:55852/kf_z/gw",
   [junit4]   2>   "node_name":"127.0.0.1:55852_kf_z%2Fgw",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 458749 INFO  (OverseerStateUpdate-96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:55852/kf_z/gw",
   [junit4]   2>   "node_name":"127.0.0.1:55852_kf_z%2Fgw",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 458749 INFO  (OverseerStateUpdate-96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 458750 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 458750 INFO  (OverseerStateUpdate-96384944917905413-127.0.0.1:55852_kf_z%2Fgw-n_0000000000) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 458751 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 458751 INFO  (zkCallback-1350-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 459751 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 459751 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 459752 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 459752 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 459752 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 459752 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/cores/collection1'
   [junit4]   2> 459752 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 459752 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 459753 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 459757 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 459761 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 459768 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 459772 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 459842 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 459846 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 459846 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 459854 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 459855 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 459856 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 459857 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 459857 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2
   [junit4]   2> 459858 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 459858 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 459858 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:45781/solr_hdfs_home
   [junit4]   2> 459858 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 459858 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/control-001/cores/collection1], dataDir=[hdfs://localhost:45781/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 459858 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@499e0159
   [junit4]   2> 459858 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:45781/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 459864 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 459864 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 459864 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 461154 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5cd5961a
   [junit4]   2> 461154 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 461154 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 461154 INFO  (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 461154 ERROR (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Error while closing
   [junit4]   2> java.lang.NullPointerException
   [junit4]   2> 	at org.apache.solr.core.SolrCore.close(SolrCore.java:1261)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:807)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:688)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:860)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:516)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 461155 ERROR (coreLoadExecutor-1020-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer Error creating core [collection1]: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
   [junit4]   2> org.apache.solr.common.SolrException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:815)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:688)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:860)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:516)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:304)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:286)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:220)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:426)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:756)
   [junit4]   2> 	... 8 more
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
   [junit4]   2> 	at java.nio.Bits.reserveMemory(Bits.java:693)
   [junit4]   2> 	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
   [junit4]   2> 	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
   [junit4]   2> 	at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:63)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:302)
   [junit4]   2> 	... 13 more
   [junit4]   2> 461156 ERROR (coreContainerWorkExecutor-1015-thread-1-processing-n:127.0.0.1:55852_kf_z%2Fgw) [n:127.0.0.1:55852_kf_z%2Fgw    ] o.a.s.c.CoreContainer Error waiting for SolrCore to be created
   [junit4]   2> java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [collection1]
   [junit4]   2> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
   [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.lambda$load$1(CoreContainer.java:544)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Unable to create core [collection1]
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:873)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:516)
   [junit4]   2> 	... 5 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:815)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:688)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:860)
   [junit4]   2> 	... 6 more
   [junit4]   2> Caused by: java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~8.388608 MB.
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:304)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:286)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:220)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:346)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:426)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:756)
   [junit4]   2> 	... 8 more
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
   [junit4]   2> 	at java.nio.Bits.reserveMemory(Bits.java:693)
   [junit4]   2> 	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
   [junit4]   2> 	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
   [junit4]   2> 	at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:63)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:302)
   [junit4]   2> 	... 13 more
   [junit4]   2> 461639 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 461641 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 461648 INFO  (zkCallback-1354-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@18b87269 name:ZooKeeperConnection Watcher:127.0.0.1:55825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 461648 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 461648 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 461648 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 461648 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 461649 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 461650 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 461714 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/shard-1-001/cores/collection1
   [junit4]   2> 461715 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/shard-1-001
   [junit4]   2> 461717 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 461726 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22c8fb1c{/kf_z/gw,null,AVAILABLE}
   [junit4]   2> 461728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.ServerConnector Started ServerConnector@446e5a2b{SSL,[ssl, http/1.1]}{127.0.0.1:58799}
   [junit4]   2> 461728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.e.j.s.Server Started @464131ms
   [junit4]   2> 461728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:45781/hdfs__localhost_45781__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/kf_z/gw, hostPort=58799, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/shard-1-001/cores}
   [junit4]   2> 461728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 461728 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/shard-1-001'
   [junit4]   2> 461729 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 461729 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 461729 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 461731 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 461732 INFO  (zkCallback-1355-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4ac706a4 name:ZooKeeperConnection Watcher:127.0.0.1:55825/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 461732 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 461732 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 461733 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 461733 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282DF373B084]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsSyncSliceTest_F83E282DF373B084-001/shard-1-001/solr.xml
   [junit4]   2> 461737 INFO  (TEST-HdfsSyncSliceTest.test-seed#[F83E282D

[...truncated too long message...]

enkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160809_043122_407.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160809_043122_407.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 255 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
ERROR: Failed to archive {README.txt=README.txt, solr/build/solr-core/test/temp/junit4-J1-20160809_043122_407.events=solr/build/solr-core/test/temp/junit4-J1-20160809_043122_407.events, heapdumps/java_pid5472.hprof=heapdumps/java_pid5472.hprof, heapdumps/java_pid5487.hprof=heapdumps/java_pid5487.hprof, solr/build/solr-core/test/temp/junit4-J0-20160809_043122_407.events=solr/build/solr-core/test/temp/junit4-J0-20160809_043122_407.events, solr/build/solr-core/test/temp/junit4-J2-20160809_043122_407.events=solr/build/solr-core/test/temp/junit4-J2-20160809_043122_407.events} due to internal error; falling back to full archiving
java.lang.NullPointerException
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1092 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1092/

13 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=18691, name=collection5, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=18691, name=collection5, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:56726: collection already exists: awholynewstresscollection_collection5_2
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:988)


FAILED:  org.apache.solr.cloud.TestHdfsCloudBackupRestore.test

Error Message:
Error from server at https://127.0.0.1:45425/solr: Collection 'hdfsbackuprestore_restored' exists, no action taken.

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:45425/solr: Collection 'hdfsbackuprestore_restored' exists, no action taken.
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC:46DD74251137C14]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:232)
	at org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:55094/vm

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:55094/vm
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:458)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 47 more


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10Minutes

Error Message:
Captured an uncaught exception in thread: Thread[id=138476, name=Thread-13265, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=138476, name=Thread-13265, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
	at java.util.HashMap.resize(HashMap.java:703)
	at java.util.HashMap.putVal(HashMap.java:628)
	at java.util.HashMap.put(HashMap.java:611)
	at java.util.HashSet.add(HashSet.java:219)
	at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:212)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:243)
	at sun.security.util.AbstractAlgorithmConstraints.checkAlgorithm(AbstractAlgorithmConstraints.java:104)
	at sun.security.util.DisabledAlgorithmConstraints.permits(DisabledAlgorithmConstraints.java:91)
	at sun.security.ssl.SSLAlgorithmConstraints.permits(SSLAlgorithmConstraints.java:144)
	at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:349)
	at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:304)
	at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:626)
	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:567)
	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1470631025828,generation=2,filelist=[_9q.fdt, _9q.fdx, _9q.fnm, _9q.nvd, _9q.nvm, _9q.si, _9q_Asserting_0.doc, _9q_Asserting_0.tim, _9q_Asserting_0.tip, _ch.fdt, _ch.fdx, _ch.fnm, _ch.nvd, _ch.nvm, _ch.si, _ch_Asserting_0.doc, _ch_Asserting_0.tim, _ch_Asserting_0.tip, _eb.fdt, _eb.fdx, _eb.fnm, _eb.nvd, _eb.nvm, _eb.si, _eb_Asserting_0.doc, _eb_Asserting_0.tim, _eb_Asserting_0.tip, _fr.cfe, _fr.cfs, _fr.si, _fy.cfe, _fy.cfs, _fy.si, _fz.cfe, _fz.cfs, _fz.si, _g0.cfe, _g0.cfs, _g0.si, _g1.cfe, _g1.cfs, _g1.si, _g2.cfe, _g2.cfs, _g2.si, _g3.cfe, _g3.cfs, _g3.si, _g4.cfe, _g4.cfs, _g4.si, _g5.cfe, _g5.cfs, _g5.si, _g6.cfe, _g6.cfs, _g6.si, _g7.cfe, _g7.cfs, _g7.si, _g8.cfe, _g8.cfs, _g8.si, segments_2]}]> but was:<[{indexVersion=1470631025828,generation=2,filelist=[_9q.fdt, _9q.fdx, _9q.fnm, _9q.nvd, _9q.nvm, _9q.si, _9q_Asserting_0.doc, _9q_Asserting_0.tim, _9q_Asserting_0.tip, _ch.fdt, _ch.fdx, _ch.fnm, _ch.nvd, _ch.nvm, _ch.si, _ch_Asserting_0.doc, _ch_Asserting_0.tim, _ch_Asserting_0.tip, _eb.fdt, _eb.fdx, _eb.fnm, _eb.nvd, _eb.nvm, _eb.si, _eb_Asserting_0.doc, _eb_Asserting_0.tim, _eb_Asserting_0.tip, _fr.cfe, _fr.cfs, _fr.si, _fy.cfe, _fy.cfs, _fy.si, _fz.cfe, _fz.cfs, _fz.si, _g0.cfe, _g0.cfs, _g0.si, _g1.cfe, _g1.cfs, _g1.si, _g2.cfe, _g2.cfs, _g2.si, _g3.cfe, _g3.cfs, _g3.si, _g4.cfe, _g4.cfs, _g4.si, _g5.cfe, _g5.cfs, _g5.si, _g6.cfe, _g6.cfs, _g6.si, _g7.cfe, _g7.cfs, _g7.si, _g8.cfe, _g8.cfs, _g8.si, segments_2]}, {indexVersion=1470631025828,generation=3,filelist=[_9q.fdt, _9q.fdx, _9q.fnm, _9q.nvd, _9q.nvm, _9q.si, _9q_Asserting_0.doc, _9q_Asserting_0.tim, _9q_Asserting_0.tip, _ch.fdt, _ch.fdx, _ch.fnm, _ch.nvd, _ch.nvm, _ch.si, _ch_Asserting_0.doc, _ch_Asserting_0.tim, _ch_Asserting_0.tip, _eb.fdt, _eb.fdx, _eb.fnm, _eb.nvd, _eb.nvm, _eb.si, _eb_Asserting_0.doc, _eb_Asserting_0.tim, _eb_Asserting_0.tip, _fr.cfe, _fr.cfs, _fr.si, _fz.cfe, _fz.cfs, _fz.si, _g9.cfe, _g9.cfs, _g9.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1470631025828,generation=2,filelist=[_9q.fdt, _9q.fdx, _9q.fnm, _9q.nvd, _9q.nvm, _9q.si, _9q_Asserting_0.doc, _9q_Asserting_0.tim, _9q_Asserting_0.tip, _ch.fdt, _ch.fdx, _ch.fnm, _ch.nvd, _ch.nvm, _ch.si, _ch_Asserting_0.doc, _ch_Asserting_0.tim, _ch_Asserting_0.tip, _eb.fdt, _eb.fdx, _eb.fnm, _eb.nvd, _eb.nvm, _eb.si, _eb_Asserting_0.doc, _eb_Asserting_0.tim, _eb_Asserting_0.tip, _fr.cfe, _fr.cfs, _fr.si, _fy.cfe, _fy.cfs, _fy.si, _fz.cfe, _fz.cfs, _fz.si, _g0.cfe, _g0.cfs, _g0.si, _g1.cfe, _g1.cfs, _g1.si, _g2.cfe, _g2.cfs, _g2.si, _g3.cfe, _g3.cfs, _g3.si, _g4.cfe, _g4.cfs, _g4.si, _g5.cfe, _g5.cfs, _g5.si, _g6.cfe, _g6.cfs, _g6.si, _g7.cfe, _g7.cfs, _g7.si, _g8.cfe, _g8.cfs, _g8.si, segments_2]}]> but was:<[{indexVersion=1470631025828,generation=2,filelist=[_9q.fdt, _9q.fdx, _9q.fnm, _9q.nvd, _9q.nvm, _9q.si, _9q_Asserting_0.doc, _9q_Asserting_0.tim, _9q_Asserting_0.tip, _ch.fdt, _ch.fdx, _ch.fnm, _ch.nvd, _ch.nvm, _ch.si, _ch_Asserting_0.doc, _ch_Asserting_0.tim, _ch_Asserting_0.tip, _eb.fdt, _eb.fdx, _eb.fnm, _eb.nvd, _eb.nvm, _eb.si, _eb_Asserting_0.doc, _eb_Asserting_0.tim, _eb_Asserting_0.tip, _fr.cfe, _fr.cfs, _fr.si, _fy.cfe, _fy.cfs, _fy.si, _fz.cfe, _fz.cfs, _fz.si, _g0.cfe, _g0.cfs, _g0.si, _g1.cfe, _g1.cfs, _g1.si, _g2.cfe, _g2.cfs, _g2.si, _g3.cfe, _g3.cfs, _g3.si, _g4.cfe, _g4.cfs, _g4.si, _g5.cfe, _g5.cfs, _g5.si, _g6.cfe, _g6.cfs, _g6.si, _g7.cfe, _g7.cfs, _g7.si, _g8.cfe, _g8.cfs, _g8.si, segments_2]}, {indexVersion=1470631025828,generation=3,filelist=[_9q.fdt, _9q.fdx, _9q.fnm, _9q.nvd, _9q.nvm, _9q.si, _9q_Asserting_0.doc, _9q_Asserting_0.tim, _9q_Asserting_0.tip, _ch.fdt, _ch.fdx, _ch.fnm, _ch.nvd, _ch.nvm, _ch.si, _ch_Asserting_0.doc, _ch_Asserting_0.tim, _ch_Asserting_0.tip, _eb.fdt, _eb.fdx, _eb.fnm, _eb.nvd, _eb.nvm, _eb.si, _eb_Asserting_0.doc, _eb_Asserting_0.tim, _eb_Asserting_0.tip, _fr.cfe, _fr.cfs, _fr.si, _fz.cfe, _fz.cfs, _fz.si, _g9.cfe, _g9.cfs, _g9.si, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC:A9EEF3A88FA71FEF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1227)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=99402, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=99442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=99407, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=99432, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=99422, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=99412, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=99447, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=99427, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=99417, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=99437, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=99402, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=99442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=99407, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=99432, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=99422, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=99412, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=99447, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=99427, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=99417, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=99437, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=11497, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11961, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=13847, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=7298, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=11960, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=11496, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=7571, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=7570, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=13844, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=7569, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=9890, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=9616, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=9614, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=7705, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=7706, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=7940, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=11962, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=10063, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=13846, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=11959, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=7943, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=10304, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=13845, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=7572, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=10064, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=7704, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=7941, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=9891, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=7707, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=9892, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=12426, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=10062, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=12425, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=7297, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=7300, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=9615, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=10061, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=11495, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=12427, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=9889, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=11494, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=10306, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=10305, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=10303, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=7942, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=7299, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=12428, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=9617, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=11497, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11961, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=13847, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=7298, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=11960, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=11496, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=7571, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=7570, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=13844, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=7569, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=9890, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=9616, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=9614, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=7705, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=7706, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=7940, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=11962, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=10063, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=13846, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=11959, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=7943, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=10304, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=13845, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=7572, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=10064, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=7704, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=7941, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=9891, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=7707, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=9892, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=12426, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=10062, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=12425, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=7297, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=7300, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=9615, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=10061, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=11495, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=12427, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=9889, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=11494, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=10306, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=10305, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=10303, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=7942, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=7299, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=12428, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=9617, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=141213, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=141864, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=141212, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=141865, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=141866, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=141210, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=141867, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=141211, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=141213, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=141864, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=141212, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=141865, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=141866, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=141210, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=141867, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=141211, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=17446, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=16669, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=16931, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=17188, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=17445, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=16445, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=16324, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=16444, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=16323, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=16670, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=16932, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=17189, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest: 
   1) Thread[id=17446, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=16669, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=16931, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=17188, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=17445, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=16445, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=16324, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=16444, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=16323, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=16670, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=16932, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=17189, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)




Build Log:
[...truncated 11022 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestHdfsCloudBackupRestore
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.TestHdfsCl

[...truncated too long message...]

ion evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    6) Thread[id=141210, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    7) Thread[id=141867, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2>    8) Thread[id=141211, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]   2> Aug 08, 2016 4:44:04 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true): {}, locale=uk, timezone=Brazil/East
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=118077824,total=502267904
   [junit4]   2> NOTE: All tests run in this JVM: [CoreMergeIndexesAdminHandlerTest, TestNamedUpdateProcessors, TestSolrCoreProperties, ChaosMonkeySafeLeaderTest, TestDocBasedVersionConstraints, BlockCacheTest, TestDynamicFieldResource, CSVRequestHandlerTest, TestIntervalFaceting, BlobRepositoryCloudTest, TestScoreJoinQPScore, TestShortCircuitedRequests, TestClusterStateMutator, TestMinMaxOnMultiValuedField, DistributedQueryComponentOptimizationTest, SchemaVersionSpecificBehaviorTest, TestCustomSort, HdfsUnloadDistributedZkTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestMaxScoreQueryParser, CollectionsAPIAsyncDistributedZkTest, TestBM25SimilarityFactory, TestWordDelimiterFilterFactory, BigEndianAscendingWordSerializerTest, HdfsChaosMonkeySafeLeaderTest, MinimalSchemaTest, TestOverriddenPrefixQueryForCustomFieldType, TestFastLRUCache, CoreSorterTest, TestFiltering, DateRangeFieldTest, BadCopyFieldTest, TestDistributedStatsComponentCardinality, DOMUtilTest, TestExactStatsCache, TestSolrDeletionPolicy2, HdfsDirectoryTest, HdfsRecoverLeaseTest, SimplePostToolTest, TestCodecSupport, BlockJoinFacetSimpleTest, MoreLikeThisHandlerTest, TestXmlQParser, TestLFUCache, TestRestManager, TestLMJelinekMercerSimilarityFactory, TestConfigSets, TestGraphTermsQParserPlugin, HighlighterTest, TestMiniSolrCloudCluster, TestQueryUtils, DeleteReplicaTest, ConfigSetsAPITest, DirectSolrSpellCheckerTest, TermsComponentTest, TestRequestForwarding, TestFileDictionaryLookup, SolrIndexConfigTest, TestSystemIdResolver, TestFieldCache, TestDistributedGrouping, DistanceUnitsTest, WordBreakSolrSpellCheckerTest, CdcrReplicationHandlerTest, CollectionsAPIDistributedZkTest, ZkCLITest, DistributedFacetPivotSmallTest, TimeZoneUtilsTest, TestStressLiveNodes, TestEmbeddedSolrServerConstructors, TermVectorComponentTest, HdfsDirectoryFactoryTest, TestSolrCloudWithKerberosAlt, TestStandardQParsers, TestSerializedLuceneMatchVersion, SortByFunctionTest, TestCopyFieldCollectionResource, NotRequiredUniqueKeyTest, AnalyticsQueryTest, TestRuleBasedAuthorizationPlugin, TestQueryTypes, TestAuthenticationFramework, TestRandomRequestDistribution, RankQueryTest, TestGeoJSONResponseWriter, TestCursorMarkWithoutUniqueKey, TestFieldCacheSort, TestNumericTerms64, TestMiniSolrCloudClusterSSL, DistributedMLTComponentTest, CollectionReloadTest, TestDFRSimilarityFactory, ConvertedLegacyTest, TestBinaryResponseWriter, SpellCheckComponentTest, SolrCLIZkUtilsTest, TestDefaultStatsCache, TestCryptoKeys, TestQuerySenderListener, BasicDistributedZkTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, TestReplicationHandler, SearchHandlerTest, PreAnalyzedFieldTest, PrimUtilsTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, TestSolrJ, TestLRUCache, TestUtils, ZkNodePropsTest, FileUtilsTest, SolrTestCaseJ4Test, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, ConnectionReuseTest, TestJettySolrRunner, ActionThrottleTest, AliasIntegrationTest, BaseCdcrDistributedZkTest, CdcrRequestHandlerTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CdcrRequestHandlerTest -Dtests.seed=8C39E898FFEF11EC -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=uk -Dtests.timezone=Brazil/East -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | CdcrRequestHandlerTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   [junit4]    >    1) Thread[id=141213, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=141864, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=141212, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=141865, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=141866, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    6) Thread[id=141210, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    7) Thread[id=141867, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    8) Thread[id=141211, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8C39E898FFEF11EC]:0)
   [junit4] Completed [461/626 (9!)] on J1 in 125.82s, 3 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 540 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160808_035544_904.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps/java_pid11251.hprof ...
   [junit4] Heap dump file created [490222450 bytes in 2.438 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 13 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160808_035544_904.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps/java_pid11256.hprof ...
   [junit4] Heap dump file created [605540714 bytes in 12.640 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 11048 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:715: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid11251.hprof
* java_pid11256.hprof

Total time: 226 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1091 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1091/

10 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
Captured an uncaught exception in thread: Thread[id=67671, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestReplicationHandler]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=67671, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestReplicationHandler]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.nio.file.NoSuchFileException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_6C232970F819B592-001/solr-instance-021/collection1/data/index.20160806203258206/_w.fdx
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)
	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
Caused by: java.nio.file.NoSuchFileException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_6C232970F819B592-001/solr-instance-021/collection1/data/index.20160806203258206/_w.fdx
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
	at java.nio.file.Files.delete(Files.java:1126)
	at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373)
	at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
	at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:721)
	at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:715)
	at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:691)
	at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:4888)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4366)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1470544307252,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}]> but was:<[{indexVersion=1470544307252,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}, {indexVersion=1470544307252,generation=3,filelist=[_5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, _6.cfe, _6.cfs, _6.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1470544307252,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}]> but was:<[{indexVersion=1470544307252,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}, {indexVersion=1470544307252,generation=3,filelist=[_5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, _6.cfe, _6.cfs, _6.si, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592:49F432408851BB91]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1227)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=58640, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=58650, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=58620, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=58655, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=58660, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=58630, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=58615, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=58635, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=58645, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=58625, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=58640, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=58650, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=58620, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=58655, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=58660, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=58630, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=58615, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=58635, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=58645, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=58625, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=73424, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=69485, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=73423, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=70827, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=69486, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=72320, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=72567, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=72991, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=72175, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=69488, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=73422, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=72566, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=74657, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=72178, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=72990, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=69487, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=72319, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=75343, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=72992, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=71909, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=71912, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=72323, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=72989, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=75342, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=74660, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=74927, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=70824, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=72176, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=70825, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=75344, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=71911, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=75096, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=75094, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=73425, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=74925, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=72568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=74659, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=75095, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=72569, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=74928, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=71910, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=74658, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=75097, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=72322, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=72177, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=74926, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=75345, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=70826, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=73424, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=69485, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=73423, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=70827, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=69486, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=72320, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=72567, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=72991, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=72175, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=69488, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=73422, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=72566, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=74657, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=72178, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=72990, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=69487, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=72319, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=75343, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=72992, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=71909, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=71912, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=72323, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=72989, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=75342, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=74660, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=74927, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=70824, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=72176, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=70825, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=75344, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=71911, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=75096, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=75094, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=73425, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=74925, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=72568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=74659, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=75095, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=72569, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=74928, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=71910, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=74658, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=75097, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=72322, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=72177, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=74926, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=75345, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=70826, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=174161, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=174160, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=173795, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=174158, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=174159, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=173794, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=173796, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=173797, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=174161, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=174160, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=173795, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=174158, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=174159, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=173794, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=173796, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=173797, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=187130, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=187131, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=186903, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=187398, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=186904, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=186784, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=186785, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=187397, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=188602, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=188863, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=188864, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=188603, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest: 
   1) Thread[id=187130, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=187131, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=186903, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=187398, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=186904, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=186784, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=186785, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=187397, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=188602, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=188863, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=188864, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=188603, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([6C232970F819B592]:0)




Build Log:
[...truncated 11189 lines...]
   [junit4] Suite: org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_6C232970F819B592-001/init-core-data-001
   [junit4]   2> 532918 INFO  (SUITE-ConcurrentDeleteAndCreateCollectionTest-seed#[6C232970F819B592]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 532919 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testConcurrentCreateAndDeleteOverTheSameConfig
   [junit4]   2> 532920 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 532920 INFO  (Thread-56036) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 532920 INFO  (Thread-56036) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 533020 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.ZkTestServer start zk server on port:48121
   [junit4]   2> 533020 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 533020 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 533023 INFO  (zkCallback-413-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6b184eb6 name:ZooKeeperConnection Watcher:127.0.0.1:48121 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 533023 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 533023 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 533023 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 533024 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/clusterprops.json
   [junit4]   2> 533027 INFO  (jetty-launcher-412-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 533031 INFO  (jetty-launcher-412-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@57e4d020{/solr,null,AVAILABLE}
   [junit4]   2> 533033 INFO  (jetty-launcher-412-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@6aa01713{SSL,[ssl, http/1.1]}{127.0.0.1:34575}
   [junit4]   2> 533033 INFO  (jetty-launcher-412-thread-1) [    ] o.e.j.s.Server Started @535381ms
   [junit4]   2> 533034 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34575}
   [junit4]   2> 533034 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 533034 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_6C232970F819B592-001/tempDir-001/node1'
   [junit4]   2> 533034 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 533034 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 533034 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 533039 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 533043 INFO  (zkCallback-414-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@37033786 name:ZooKeeperConnection Watcher:127.0.0.1:48121/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 533043 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 533043 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 533044 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 533048 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_6C232970F819B592-001/tempDir-001/node1/.
   [junit4]   2> 533048 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 422195111
   [junit4]   2> 533048 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_6C232970F819B592-001/tempDir-001/node1]
   [junit4]   2> 533048 WARN  (jetty-launcher-412-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_6C232970F819B592-001/tempDir-001/node1/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_6C232970F819B592-001/tempDir-001/node1/lib
   [junit4]   2> 533048 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 533055 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.u.UpdateShardHandler Created UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&maxConnections=100000&maxConnectionsPerHost=100000
   [junit4]   2> 533055 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 533055 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 533055 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48121/solr
   [junit4]   2> 533055 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 533055 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 533055 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 533059 INFO  (zkCallback-417-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@461e04e5 name:ZooKeeperConnection Watcher:127.0.0.1:48121 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 533059 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 533059 INFO  (jetty-launcher-412-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 533060 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$88/1482060475@515d99d4
   [junit4]   2> 533061 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 533061 INFO  (zkCallback-418-thread-1-processing-n:127.0.0.1:34575_solr) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@672cc1f name:ZooKeeperConnection Watcher:127.0.0.1:48121/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 533061 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 533062 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 533063 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 533064 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 533065 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 533066 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 533066 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 533067 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 533068 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 533068 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 533069 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 533069 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 533070 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 533070 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 533071 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 533072 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 533072 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 533072 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96373541778292739-127.0.0.1:34575_solr-n_0000000000
   [junit4]   2> 533073 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34575_solr
   [junit4]   2> 533073 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 533073 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.Overseer Overseer (id=96373541778292739-127.0.0.1:34575_solr-n_0000000000) starting
   [junit4]   2> 533074 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 533077 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 533077 INFO  (OverseerCollectionConfigSetProcessor-96373541778292739-127.0.0.1:34575_solr-n_0000000000) [n:127.0.0.1:34575_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 533077 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34575_solr
   [junit4]   2> 533077 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:34575_solr
   [junit4]   2> 533077 INFO  (OverseerStateUpdate-96373541778292739-127.0.0.1:34575_solr-n_0000000000) [n:127.0.0.1:34575_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 533079 INFO  (zkCallback-418-thread-1-processing-n:127.0.0.1:34575_solr) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 533084 INFO  (OverseerStateUpdate-96373541778292739-127.0.0.1:34575_solr-n_0000000000) [n:127.0.0.1:34575_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 533104 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 533104 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 533105 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.cloud.ConcurrentDeleteAndCreateCollectionTest_6C232970F819B592-001/tempDir-001/node1/.
   [junit4]   2> 533105 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 533106 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0
   [junit4]   2> 533106 INFO  (jetty-launcher-412-thread-1) [n:127.0.0.1:34575_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 533106 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 533106 INFO  (TEST-ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteOverTheSameConfig-seed#[6C232970F819B592]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 533107 INFO  (zkCallback-420-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@aa30167 name:ZooKeeperConnection Watcher:127.0.0.1:48121 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4] 

[...truncated too long message...]

enkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160807_041009_751.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160807_041009_751.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Forked process returned with error code: 134. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160807_041009_751.sysout
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1481)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: At least one slave process threw an exception, first: Forked process returned with error code: 134. Very likely a JVM crash.  See process stdout at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160807_041009_751.sysout

Total time: 230 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
ERROR: Failed to archive {README.txt=README.txt, heapdumps/java_pid23867.hprof=heapdumps/java_pid23867.hprof, solr/build/solr-core/test/J2/hs_err_pid23867.log=solr/build/solr-core/test/J2/hs_err_pid23867.log, solr/build/solr-core/test/temp/junit4-J2-20160807_041009_751.events=solr/build/solr-core/test/temp/junit4-J2-20160807_041009_751.events, heapdumps/java_pid23858.hprof=heapdumps/java_pid23858.hprof, solr/build/solr-core/test/temp/junit4-J0-20160807_041009_749.events=solr/build/solr-core/test/temp/junit4-J0-20160807_041009_749.events, solr/build/solr-core/test/temp/junit4-J1-20160807_041009_749.events=solr/build/solr-core/test/temp/junit4-J1-20160807_041009_749.events} due to internal error; falling back to full archiving
java.lang.NullPointerException
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1090 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1090/

12 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=36751, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=36751, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34961/s_jmz: collection already exists: awholynewstresscollection_collection0_1
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1620)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:988)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:51660/x_oh/tw

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:51660/x_oh/tw
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:458)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 47 more


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1470459476879,generation=2,filelist=[_7p.cfe, _7p.cfs, _7p.si, _7w.fdt, _7w.fdx, _7w.fnm, _7w.nvd, _7w.nvm, _7w.si, _7w_FST50_0.doc, _7w_FST50_0.tfp, _7x.cfe, _7x.cfs, _7x.si, _7y.fdt, _7y.fdx, _7y.fnm, _7y.nvd, _7y.nvm, _7y.si, _7y_FST50_0.doc, _7y_FST50_0.tfp, _81.fdt, _81.fdx, _81.fnm, _81.nvd, _81.nvm, _81.si, _81_FST50_0.doc, _81_FST50_0.tfp, _82.cfe, _82.cfs, _82.si, _83.fdt, _83.fdx, _83.fnm, _83.nvd, _83.nvm, _83.si, _83_FST50_0.doc, _83_FST50_0.tfp, _84.fdt, _84.fdx, _84.fnm, _84.nvd, _84.nvm, _84.si, _84_FST50_0.doc, _84_FST50_0.tfp, _85.fdt, _85.fdx, _85.fnm, _85.nvd, _85.nvm, _85.si, _85_FST50_0.doc, _85_FST50_0.tfp, _86.cfe, _86.cfs, _86.si, _87.fdt, _87.fdx, _87.fnm, _87.nvd, _87.nvm, _87.si, _87_FST50_0.doc, _87_FST50_0.tfp, segments_2]}]> but was:<[{indexVersion=1470459476879,generation=2,filelist=[_7p.cfe, _7p.cfs, _7p.si, _7w.fdt, _7w.fdx, _7w.fnm, _7w.nvd, _7w.nvm, _7w.si, _7w_FST50_0.doc, _7w_FST50_0.tfp, _7x.cfe, _7x.cfs, _7x.si, _7y.fdt, _7y.fdx, _7y.fnm, _7y.nvd, _7y.nvm, _7y.si, _7y_FST50_0.doc, _7y_FST50_0.tfp, _81.fdt, _81.fdx, _81.fnm, _81.nvd, _81.nvm, _81.si, _81_FST50_0.doc, _81_FST50_0.tfp, _82.cfe, _82.cfs, _82.si, _83.fdt, _83.fdx, _83.fnm, _83.nvd, _83.nvm, _83.si, _83_FST50_0.doc, _83_FST50_0.tfp, _84.fdt, _84.fdx, _84.fnm, _84.nvd, _84.nvm, _84.si, _84_FST50_0.doc, _84_FST50_0.tfp, _85.fdt, _85.fdx, _85.fnm, _85.nvd, _85.nvm, _85.si, _85_FST50_0.doc, _85_FST50_0.tfp, _86.cfe, _86.cfs, _86.si, _87.fdt, _87.fdx, _87.fnm, _87.nvd, _87.nvm, _87.si, _87_FST50_0.doc, _87_FST50_0.tfp, segments_2]}, {indexVersion=1470459476879,generation=3,filelist=[_7p.cfe, _7p.cfs, _7p.si, _83.fdt, _83.fdx, _83.fnm, _83.nvd, _83.nvm, _83.si, _83_FST50_0.doc, _83_FST50_0.tfp, _87.fdt, _87.fdx, _87.fnm, _87.nvd, _87.nvm, _87.si, _87_FST50_0.doc, _87_FST50_0.tfp, _88.fdt, _88.fdx, _88.fnm, _88.nvd, _88.nvm, _88.si, _88_FST50_0.doc, _88_FST50_0.tfp, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1470459476879,generation=2,filelist=[_7p.cfe, _7p.cfs, _7p.si, _7w.fdt, _7w.fdx, _7w.fnm, _7w.nvd, _7w.nvm, _7w.si, _7w_FST50_0.doc, _7w_FST50_0.tfp, _7x.cfe, _7x.cfs, _7x.si, _7y.fdt, _7y.fdx, _7y.fnm, _7y.nvd, _7y.nvm, _7y.si, _7y_FST50_0.doc, _7y_FST50_0.tfp, _81.fdt, _81.fdx, _81.fnm, _81.nvd, _81.nvm, _81.si, _81_FST50_0.doc, _81_FST50_0.tfp, _82.cfe, _82.cfs, _82.si, _83.fdt, _83.fdx, _83.fnm, _83.nvd, _83.nvm, _83.si, _83_FST50_0.doc, _83_FST50_0.tfp, _84.fdt, _84.fdx, _84.fnm, _84.nvd, _84.nvm, _84.si, _84_FST50_0.doc, _84_FST50_0.tfp, _85.fdt, _85.fdx, _85.fnm, _85.nvd, _85.nvm, _85.si, _85_FST50_0.doc, _85_FST50_0.tfp, _86.cfe, _86.cfs, _86.si, _87.fdt, _87.fdx, _87.fnm, _87.nvd, _87.nvm, _87.si, _87_FST50_0.doc, _87_FST50_0.tfp, segments_2]}]> but was:<[{indexVersion=1470459476879,generation=2,filelist=[_7p.cfe, _7p.cfs, _7p.si, _7w.fdt, _7w.fdx, _7w.fnm, _7w.nvd, _7w.nvm, _7w.si, _7w_FST50_0.doc, _7w_FST50_0.tfp, _7x.cfe, _7x.cfs, _7x.si, _7y.fdt, _7y.fdx, _7y.fnm, _7y.nvd, _7y.nvm, _7y.si, _7y_FST50_0.doc, _7y_FST50_0.tfp, _81.fdt, _81.fdx, _81.fnm, _81.nvd, _81.nvm, _81.si, _81_FST50_0.doc, _81_FST50_0.tfp, _82.cfe, _82.cfs, _82.si, _83.fdt, _83.fdx, _83.fnm, _83.nvd, _83.nvm, _83.si, _83_FST50_0.doc, _83_FST50_0.tfp, _84.fdt, _84.fdx, _84.fnm, _84.nvd, _84.nvm, _84.si, _84_FST50_0.doc, _84_FST50_0.tfp, _85.fdt, _85.fdx, _85.fnm, _85.nvd, _85.nvm, _85.si, _85_FST50_0.doc, _85_FST50_0.tfp, _86.cfe, _86.cfs, _86.si, _87.fdt, _87.fdx, _87.fnm, _87.nvd, _87.nvm, _87.si, _87_FST50_0.doc, _87_FST50_0.tfp, segments_2]}, {indexVersion=1470459476879,generation=3,filelist=[_7p.cfe, _7p.cfs, _7p.si, _83.fdt, _83.fdx, _83.fnm, _83.nvd, _83.nvm, _83.si, _83_FST50_0.doc, _83_FST50_0.tfp, _87.fdt, _87.fdx, _87.fnm, _87.nvd, _87.nvm, _87.si, _87_FST50_0.doc, _87_FST50_0.tfp, _88.fdt, _88.fdx, _88.fnm, _88.nvd, _88.nvm, _88.si, _88_FST50_0.doc, _88_FST50_0.tfp, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4:D6CE2DAC92CD0CF7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1227)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:50380/forceleader_test_collection_shard1_replica1]

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://127.0.0.1:50380/forceleader_test_collection_shard1_replica1]
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4:158E025CDB07FB95]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:760)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
	at org.apache.solr.cloud.ForceLeaderTest.sendDoc(ForceLeaderTest.java:424)
	at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:50380/forceleader_test_collection_shard1_replica1]
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:393)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:735)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:50380/forceleader_test_collection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:615)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:382)
	... 6 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:50380 [/127.0.0.1] failed: Connection refused
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 10 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	... 20 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=51062, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=51082, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=51102, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=51057, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=51092, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=51077, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=51067, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=51097, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=51072, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=51087, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=51062, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=51082, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=51102, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=51057, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=51092, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=51077, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=51067, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=51097, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=51072, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=51087, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=75936, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=82541, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=75669, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=75670, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=79399, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=76350, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=76106, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=82919, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=82275, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=82276, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=75207, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=78034, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=78035, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=82680, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=75938, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=75937, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=74752, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=82544, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=76107, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=82543, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=76347, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=76348, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=74755, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=79401, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=75208, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=82274, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=75671, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=75205, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=76108, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=82918, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=74753, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=82916, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=82542, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=82681, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=75935, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=82679, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=74754, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=82272, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=75667, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=79398, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=75206, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=76349, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=79400, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=82678, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=78036, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=82917, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=78037, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=76105, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=75936, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=82541, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=75669, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=75670, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=79399, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=76350, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=76106, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=82919, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=82275, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=82276, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=75207, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=78034, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=78035, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=82680, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=75938, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=75937, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=74752, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=82544, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=76107, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=82543, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=76347, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=76348, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=74755, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=79401, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=75208, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=82274, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=75671, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=75205, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=76108, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=82918, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=74753, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=82916, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=82542, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=82681, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=75935, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=82679, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=74754, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=82272, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=75667, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=79398, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=75206, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=76349, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=79400, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=82678, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=78036, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=82917, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=78037, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=76105, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=49793, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=49792, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=49794, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=50137, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=49791, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=50135, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=50134, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=50136, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=49793, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=49792, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=49794, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=50137, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=49791, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=50135, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=50134, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=50136, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=48809, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=48927, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=48546, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=49167, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=49426, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=47219, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=49427, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=48547, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=49168, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=47218, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=48810, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=48926, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest: 
   1) Thread[id=48809, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=48927, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=48546, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=49167, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=49426, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=47219, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=49427, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=48547, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=49168, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=47218, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=48810, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=48926, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([F319369CE28502F4]:0)




Build Log:
[...truncated 11540 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_F319369CE28502F4-001/init-core-data-001
   [junit4]   2> 620052 INFO  (SUITE-TestReplicationHandler-seed#[F319369CE28502F4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 620054 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterWrite2Slave-seed#[F319369CE28502F4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestReplicateAfterWrite2Slave
   [j

[...truncated too long message...]

ns/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160806_044452_886.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160806_044452_886.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 226 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
ERROR: Failed to archive {heapdumps/java_pid11155.hprof=heapdumps/java_pid11155.hprof, heapdumps/java_pid11156.hprof=heapdumps/java_pid11156.hprof, solr/build/solr-core/test/temp/junit4-J0-20160806_044452_886.events=solr/build/solr-core/test/temp/junit4-J0-20160806_044452_886.events, README.txt=README.txt, solr/build/solr-core/test/temp/junit4-J2-20160806_044452_886.events=solr/build/solr-core/test/temp/junit4-J2-20160806_044452_886.events, solr/build/solr-core/test/temp/junit4-J1-20160806_044452_886.events=solr/build/solr-core/test/temp/junit4-J1-20160806_044452_886.events} due to internal error; falling back to full archiving
java.lang.NullPointerException
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1089 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1089/

16 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:50532/forceleader_test_collection_shard1_replica2]

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://127.0.0.1:50532/forceleader_test_collection_shard1_replica2]
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479:1BFF098003958D18]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:760)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
	at org.apache.solr.cloud.ForceLeaderTest.sendDoc(ForceLeaderTest.java:424)
	at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:50532/forceleader_test_collection_shard1_replica2]
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:393)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:735)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:50532/forceleader_test_collection_shard1_replica2
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:615)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:382)
	... 6 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:50532 [/127.0.0.1] failed: Connection refused
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 10 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	... 20 more


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479:753C029A94EB1981]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:309)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:127)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
expected:<0> but was:<28>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<28>
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479:753C029A94EB1981]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:146)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 4 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.security.BasicAuthIntegrationTest.testBasics

Error Message:
IOException occured when talking to server at: http://127.0.0.1:44802/solr/testSolrCloudCollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException occured when talking to server at: http://127.0.0.1:44802/solr/testSolrCloudCollection_shard1_replica1
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479:C0B0936C02F92A09]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:760)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.security.BasicAuthIntegrationTest.doExtraTests(BasicAuthIntegrationTest.java:193)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testCollectionCreateSearchDelete(TestMiniSolrCloudClusterBase.java:196)
	at org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testBasics(TestMiniSolrCloudClusterBase.java:79)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:44802/solr/testSolrCloudCollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:622)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:735)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:44802 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 10 more


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10Minutes

Error Message:
Captured an uncaught exception in thread: Thread[id=71289, name=Thread-62376, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=71289, name=Thread-62376, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)
	at sun.security.ssl.HandshakeMessage$ServerHello.<init>(HandshakeMessage.java:347)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:206)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:190)
	at org.apache.solr.core.OneIndexer.run(OpenCloseCoreStressTest.java:400)


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test15Seconds

Error Message:
Core 00003_core bad! expected:<467> but was:<464>

Stack Trace:
java.lang.AssertionError: Core 00003_core bad! expected:<467> but was:<464>
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479:7526D182A7A3785]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.OpenCloseCoreStressTest.checkResults(OpenCloseCoreStressTest.java:283)
	at org.apache.solr.core.OpenCloseCoreStressTest.doStress(OpenCloseCoreStressTest.java:182)
	at org.apache.solr.core.OpenCloseCoreStressTest.test15Seconds(OpenCloseCoreStressTest.java:116)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=52404, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=52429, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=52394, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=52414, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=52424, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=52409, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=52419, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=52389, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=52434, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=52399, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=52404, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=52429, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=52394, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=52414, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=52424, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=52409, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=52419, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=52389, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=52434, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=52399, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=84874, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=90358, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=88189, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=90357, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=90359, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=88328, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=88564, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=81993, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=81756, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=87913, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=90827, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=81579, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=88565, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=90824, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=83522, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=81582, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=81758, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=88566, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=81996, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=81755, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=81580, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=84876, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=88188, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=88327, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=90826, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=81994, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=87912, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=88563, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=83525, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=88187, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=81581, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=90356, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=83523, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=83524, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=87910, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=90825, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=81311, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=81314, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=88326, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=84877, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=88186, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=81313, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=81757, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=81312, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=81995, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=84875, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=88329, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=87911, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=84874, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=90358, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=88189, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=90357, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=90359, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=88328, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=88564, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=81993, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=81756, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=87913, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=90827, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=81579, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=88565, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=90824, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=83522, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=81582, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=81758, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=88566, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=81996, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=81755, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=81580, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=84876, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=88188, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=88327, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=90826, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=81994, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=87912, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=88563, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=83525, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=88187, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=81581, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=90356, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=83523, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=83524, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=87910, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=90825, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=81311, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=81314, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=88326, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=84877, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=88186, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=81313, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=81757, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=81312, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=81995, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=84875, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=88329, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=87911, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=5514, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=5516, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=5848, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=5850, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=5513, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=5849, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=5847, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=5515, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=5514, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=5516, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=5848, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=5850, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=5513, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=5849, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=5847, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=5515, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([FD683D403A177479]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.car

[...truncated too long message...]

lr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160805_110644_194.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160805_110644_194.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 247 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
ERROR: Failed to archive {README.txt=README.txt, heapdumps/java_pid20212.hprof=heapdumps/java_pid20212.hprof, heapdumps/java_pid20219.hprof=heapdumps/java_pid20219.hprof, solr/build/solr-core/test/temp/junit4-J0-20160805_110644_193.events=solr/build/solr-core/test/temp/junit4-J0-20160805_110644_193.events, heapdumps/java_pid20213.hprof=heapdumps/java_pid20213.hprof, solr/build/solr-core/test/temp/junit4-J1-20160805_110644_193.events=solr/build/solr-core/test/temp/junit4-J1-20160805_110644_193.events, solr/build/solr-core/test/temp/junit4-J2-20160805_110644_194.events=solr/build/solr-core/test/temp/junit4-J2-20160805_110644_194.events} due to internal error; falling back to full archiving
java.lang.NullPointerException
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1088 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1088/

18 tests failed.
FAILED:  org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.testProxyValidateHost

Error Message:
Error from server at https://127.0.0.1:39539/solr: Expected mime type application/octet-stream but got application/json. {   "RemoteException" : {     "message" : "Unauthorized connection for super-user: localHostAnyGroup from IP localhost",     "exception" : "AuthorizationException",     "javaClassName" : "org.apache.hadoop.security.authorize.AuthorizationException"   } }

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:39539/solr: Expected mime type application/octet-stream but got application/json. {
  "RemoteException" : {
    "message" : "Unauthorized connection for super-user: localHostAnyGroup from IP localhost",
    "exception" : "AuthorizationException",
    "javaClassName" : "org.apache.hadoop.security.authorize.AuthorizationException"
  }
}
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7:D7A62A14715F4550]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:576)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.testProxyValidateHost(TestSolrCloudWithSecureImpersonation.java:260)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAddsViaShard2LeaderClient

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7:7793885C6450289B]:0)
	at java.util.HashMap$KeySet.iterator(HashMap.java:912)
	at java.util.HashSet.iterator(HashSet.java:172)
	at sun.security.util.AbstractAlgorithmConstraints.checkAlgorithm(AbstractAlgorithmConstraints.java:108)
	at sun.security.util.DisabledAlgorithmConstraints.permits(DisabledAlgorithmConstraints.java:91)
	at sun.security.ssl.SSLAlgorithmConstraints.permits(SSLAlgorithmConstraints.java:144)
	at sun.security.ssl.SignatureAndHashAlgorithm.getSupportedAlgorithms(SignatureAndHashAlgorithm.java:158)
	at sun.security.ssl.Handshaker.getLocalSupportedSignAlgs(Handshaker.java:442)
	at sun.security.ssl.ClientHandshaker.getKickstartMessage(ClientHandshaker.java:1392)
	at sun.security.ssl.Handshaker.kickstart(Handshaker.java:1014)
	at sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1487)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1351)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testAddsMixedWithDeletesViaShard2LeaderClient

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
There were too many update fails (61 > 60) - we expect it can happen, but shouldn't easily

Stack Trace:
java.lang.AssertionError: There were too many update fails (61 > 60) - we expect it can happen, but shouldn't easily
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7:BA0CD53FDAE47C1F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:221)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Could not find collection:delete_data_dir

Stack Trace:
java.lang.AssertionError: Could not find collection:delete_data_dir
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7:BA0CD53FDAE47C1F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:134)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:159)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7:DB0251DDEA81814F]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:782)
	at org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir(TestArbitraryIndexDir.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=*[count(//doc)=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=id:2&qt=standard&start=0&rows=20&version=2.2
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:775)
	... 41 more


FAILED:  org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.test

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10Minutes

Error Message:
Captured an uncaught exception in thread: Thread[id=91015, name=Thread-81204, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=91015, name=Thread-81204, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test15Seconds

Error Message:
Captured an uncaught exception in thread: Thread[id=91112, name=Thread-81251, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=91112, name=Thread-81251, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:48783/_

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:48783/_
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor152.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=102780, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=102775, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=102785, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=102805, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=102810, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=102800, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=102815, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=102795, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=102820, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=102790, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=102780, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=102775, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=102785, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=102805, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=102810, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=102800, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=102815, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=102795, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=102820, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=102790, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=11484, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=6579, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11483, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=11243, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=10376, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=11101, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=12365, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=9961, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=12368, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=11245, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=6581, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=7956, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=10374, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=9963, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=10375, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=9693, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=9692, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=11902, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=9691, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=10822, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=10823, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=11482, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=12366, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=11242, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=10138, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=6580, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=10377, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=10137, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=10824, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=11100, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=11244, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=10821, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=11485, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=12367, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=10140, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=9962, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=11098, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=11099, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=6582, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=10139, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=11903, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=11904, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=11901, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=7957, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=9960, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=7959, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=7958, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=9690, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=11484, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=6579, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=11483, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=11243, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=10376, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=11101, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=12365, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=9961, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=12368, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=11245, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=6581, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=7956, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=10374, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=9963, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=10375, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=9693, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=9692, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=11902, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=9691, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=10822, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=10823, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=11482, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=12366, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=11242, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=10138, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=6580, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=10377, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=10137, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=10824, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=11100, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=11244, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=10821, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=11485, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=12367, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=10140, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=9962, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=11098, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=11099, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=6582, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=10139, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=11903, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=11904, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=11901, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=7957, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=9960, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=7959, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=7958, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=9690, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=100760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=100402, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=100759, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=100403, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=100405, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=100404, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=100762, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=100761, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=100760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=100402, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=100759, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=100403, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=100405, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=100404, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=100762, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=100761, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([3258EAE5741811E7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=102617, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=102046, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=101288, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=101287, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=102045, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=101533, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=102373, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=101532, name=Connection evictor, state=TIME

[...truncated too long message...]

ter/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J0-20160804_042126_407.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J0-20160804_042126_407.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 230 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
ERROR: Failed to archive {README.txt=README.txt, solr/build/solr-core/test/temp/junit4-J2-20160804_042126_415.events=solr/build/solr-core/test/temp/junit4-J2-20160804_042126_415.events, heapdumps/java_pid7771.hprof=heapdumps/java_pid7771.hprof, heapdumps/java_pid7777.hprof=heapdumps/java_pid7777.hprof, solr/build/solr-core/test/temp/junit4-J0-20160804_042126_407.events=solr/build/solr-core/test/temp/junit4-J0-20160804_042126_407.events, heapdumps/java_pid7787.hprof=heapdumps/java_pid7787.hprof, solr/build/solr-core/test/temp/junit4-J1-20160804_042126_407.events=solr/build/solr-core/test/temp/junit4-J1-20160804_042126_407.events} due to internal error; falling back to full archiving
java.lang.NullPointerException
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1087 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1087/

12 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.util.TestSolrCLIRunExample

Error Message:
ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper]
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test15Seconds

Error Message:
Core 00003_core bad! expected:<503> but was:<0>

Stack Trace:
java.lang.AssertionError: Core 00003_core bad! expected:<503> but was:<0>
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D:B6D1A50D776CA971]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.core.OpenCloseCoreStressTest.checkResults(OpenCloseCoreStressTest.java:283)
	at org.apache.solr.core.OpenCloseCoreStressTest.doStress(OpenCloseCoreStressTest.java:182)
	at org.apache.solr.core.OpenCloseCoreStressTest.test15Seconds(OpenCloseCoreStressTest.java:116)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10Minutes

Error Message:
Captured an uncaught exception in thread: Thread[id=8748, name=Thread-5191, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=8748, name=Thread-5191, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)
	at java.util.HashMap.newNode(HashMap.java:1734)
	at java.util.HashMap.putVal(HashMap.java:630)
	at java.util.HashMap.put(HashMap.java:611)
	at java.util.HashSet.add(HashSet.java:219)
	at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:222)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:243)
	at sun.security.util.AbstractAlgorithmConstraints.checkAlgorithm(AbstractAlgorithmConstraints.java:104)
	at sun.security.util.DisabledAlgorithmConstraints.permits(DisabledAlgorithmConstraints.java:91)
	at sun.security.ssl.SSLAlgorithmConstraints.permits(SSLAlgorithmConstraints.java:144)
	at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:349)
	at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:304)
	at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:626)
	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:567)
	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:59607/r_dmp

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:59607/r_dmp
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=132858, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=132888, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=132873, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=132853, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=132878, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=132883, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=132863, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=132893, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=132898, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=132868, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=132858, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=132888, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=132873, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=132853, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=132878, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=132883, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=132863, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=132893, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=132898, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=132868, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=47231, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=44757, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=45199, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=44759, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=48293, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=47504, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=47506, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=47876, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=43095, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=49714, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=43093, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=45201, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=47636, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=43096, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=47638, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=49713, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=45028, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=45030, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=45440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=43523, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=48294, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=48292, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=44758, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=45029, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=47507, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=49712, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=43522, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=45441, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=43525, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=47639, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=47637, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=48291, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=47505, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=44760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=45198, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=47229, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=49715, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=43094, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=47875, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=45027, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=47232, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=47877, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=47230, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=45443, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=43524, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=45442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=47874, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=45200, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=47231, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=44757, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=45199, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=44759, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=48293, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=47504, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=47506, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=47876, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=43095, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=49714, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=43093, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=45201, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=47636, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=43096, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=47638, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=49713, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=45028, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=45030, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=45440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=43523, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=48294, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=48292, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=44758, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=45029, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=47507, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=49712, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=43522, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=45441, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=43525, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=47639, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=47637, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=48291, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=47505, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=44760, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=45198, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=47229, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=49715, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=43094, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=47875, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=45027, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=47232, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=47877, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=47230, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=45443, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=43524, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=45442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=47874, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=45200, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=10818, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11149, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11152, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=10820, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=11151, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=10819, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=10821, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=11150, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=10818, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11149, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=11152, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=10820, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=11151, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=10819, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=10821, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=11150, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=120777, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=121042, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=120282, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=120776, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=119893, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=119894, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=121041, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=120165, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=120166, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=120514, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=120283, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=120513, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest: 
   1) Thread[id=120777, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=121042, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=120282, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=120776, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=119893, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=119894, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=121041, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=120165, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=120166, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=120514, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=120283, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=120513, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4CEBF5556701EA8D]:0)




Build Log:
[...truncated 11547 lines...]
   [junit4] Suite: org.apache.solr.cloud.CdcrRequestHandlerTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest_4CEBF5556701EA8D-001/init-core-data-001
   [junit4]   2> 925103 INFO  (SUITE-CdcrRequestHandlerTest-seed#[4CEBF5556701EA8D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 925103 INFO  (SUITE-CdcrRequestHandlerTest-seed#[4CEBF5556701EA8D]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /f/t
   [junit4]   2> 925105 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 925106 INFO  (Thread-3891) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 925106 INFO  (Thread-3891) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 925206 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.ZkTestServer start zk server on port:38682
   [junit4]   2> 925206 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 925206 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 925208 INFO  (zkCallback-1262-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6040a440 name:ZooKeeperConnection Watcher:127.0.0.1:38682 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 925208 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 925208 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 925208 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 925210 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 925210 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 925211 INFO  (zkCallback-1263-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@26137bcb name:ZooKeeperConnection Watcher:127.0.0.1:38682/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 925211 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 925211 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 925211 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 925212 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 925213 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 925213 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 925214 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 925214 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 925215 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 925215 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 925216 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 925216 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 925217 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 925217 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 925218 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 925218 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 925219 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 925219 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 925220 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 925220 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 925221 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 925221 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 925221 INFO  (TEST-CdcrRequestHandlerTest.testCheckpointActions-seed#[4CEBF5556701EA8D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Luc

[...truncated too long message...]

not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206258 INFO  (qtp1460371892-154526) [n:127.0.0.1:53455_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection4_0_shard1_replica4 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206258 INFO  (qtp1350247004-154430) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection2_0_shard3_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206258 INFO  (qtp1460371892-151384) [n:127.0.0.1:53455_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection3_0_shard1_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206258 INFO  (qtp1460371892-154527) [n:127.0.0.1:53455_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection5_0_shard3_replica4 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206258 INFO  (qtp1460371892-154505) [n:127.0.0.1:53455_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection5_0_shard2_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1460371892-154486) [n:127.0.0.1:53455_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection3_0_shard2_replica4 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-152733) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard3_replica2 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-154431) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection4_0_shard3_replica2 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-154537) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection2_0_shard2_replica2 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-152754) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection5_0_shard3_replica1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1460371892-154467) [n:127.0.0.1:53455_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard1_replica2 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-154409) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection5_0_shard1_replica4 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-152731) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection4_0_shard2_replica1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-154444) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection3_0_shard1_replica1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=ClassicSimilarity, locale=ar, timezone=CAT
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=637,free=60597080,total=477102080
   [junit4]   2> NOTE: All tests run in this JVM: [TestRecoveryHdfs, EnumFieldTest, AnalysisAfterCoreReloadTest, TestImplicitCoreProperties, TestReqParamsAPI, TestDocTermOrds, CursorPagingTest, TestSolrFieldCacheMBean, HighlighterMaxOffsetTest, ZkNodePropsTest, ActionThrottleTest, TestFaceting, TestHighFrequencyDictionaryFactory, HdfsChaosMonkeyNothingIsSafeTest, DistributedQueryComponentCustomSortTest, RulesTest, HdfsRestartWhileUpdatingTest, TestSolrCloudWithDelegationTokens, DeleteLastCustomShardedReplicaTest, TestJoin, LeaderInitiatedRecoveryOnCommitTest, TestRemoteStreaming, StressHdfsTest, TestFilteredDocIdSet, TestAddFieldRealTimeGet, SolrCoreTest, TestPivotHelperCode, CoreAdminRequestStatusTest, TestDFISimilarityFactory, BigEndianAscendingWordDeserializerTest, OverseerCollectionConfigSetProcessorTest, DistributedExpandComponentTest, SynonymTokenizerTest, SuggestComponentTest, StandardRequestHandlerTest, ReturnFieldsTest, TestPhraseSuggestions, TestRandomDVFaceting, TestSolrConfigHandlerCloud, TestBackupRepositoryFactory, AlternateDirectoryTest, TestNoOpRegenerator, VMParamsZkACLAndCredentialsProvidersTest, ResourceLoaderTest, TestDynamicFieldCollectionResource, TestSSLRandomization, TestCustomDocTransformer, SecurityConfHandlerTest, ConnectionReuseTest, TestCollationFieldDocValues, TestSortByMinMaxFunction, PolyFieldTest, CdcrReplicationDistributedZkTest, TestScoreJoinQPNoScore, IndexSchemaRuntimeFieldTest, AsyncCallRequestStatusResponseTest, ResponseLogComponentTest, TestReplicaProperties, PreAnalyzedUpdateProcessorTest, TestDownShardTolerantSearch, TestDocSet, SystemInfoHandlerTest, CachingDirectoryFactoryTest, CurrencyFieldOpenExchangeTest, TestCSVLoader, TestNumericTerms32, SignatureUpdateProcessorFactoryTest, TestComplexPhraseQParserPlugin, CollectionTooManyReplicasTest, TestArbitraryIndexDir, TestConfigReload, TestFieldCacheVsDocValues, TestBlendedInfixSuggestions, TestSubQueryTransformerCrossCore, TestLMDirichletSimilarityFactory, TestLeaderElectionZkExpiry, JSONWriterTest, DistributedQueryElevationComponentTest, SOLR749Test, TestBinaryField, AddBlockUpdateTest, ExternalFileFieldSortTest, UUIDFieldTest, FileBasedSpellCheckerTest, TestSchemaManager, CacheHeaderTest, RemoteQueryErrorTest, SyncSliceTest, LeaderElectionIntegrationTest, ChangedSchemaMergeTest, OutputWriterTest, ForceLeaderTest, TestFunctionQuery, DataDrivenBlockJoinTest, ShardSplitTest, CheckHdfsIndexTest, TestPseudoReturnFields, TestTolerantUpdateProcessorRandomCloud, ZkStateWriterTest, TestFoldingMultitermQuery, RAMDirectoryFactoryTest, ExitableDirectoryReaderTest, TestSimpleQParserPlugin, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, TestDistributedSearch, SolrCoreCheckLockOnStartupTest, TestAtomicUpdateErrorCases, QueryEqualityTest, DefaultValueUpdateProcessorTest, TestSolrDeletionPolicy1, DebugComponentTest, TestSurroundQueryParser, LukeRequestHandlerTest, TestOmitPositions, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, SolrPluginUtilsTest, UpdateRequestProcessorFactoryTest, TestAnalyzedSuggestions, JsonLoaderTest, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, TestQuerySenderNoQuery, CopyFieldTest, TestStressRecovery, TestMergePolicyConfig, TestSolrDeletionPolicy2, SampleTest, TestSearchPerf, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, SpellPossibilityIteratorTest, TestCharFilters, TestXIncludeConfig, EchoParamsTest, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestJmxMonitoredMap, TimeZoneUtilsTest, TestFastOutputStream, URLClassifyProcessorTest, PluginInfoTest, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, DeleteReplicaTest, DistribJoinFromCollectionTest, MultiThreadedOCPTest, TestRandomFlRTGCloud, TestSolrCloudWithSecureImpersonation, TestTolerantUpdateProcessorCloud, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2> 11206672 INFO  (qtp1350247004-154434) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard2_replica1 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp238315660-154522) [n:127.0.0.1:57735_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection2_0_shard1_replica2 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-154535) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection3_0_shard3_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1460371892-154526) [n:127.0.0.1:53455_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection4_0_shard1_replica4 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 11206672 INFO  (qtp1350247004-154430) [n:127.0.0.1:57086_    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection2_0_shard3_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4] ERROR   0.00s J0 | HdfsCollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4] Completed [626/626 (8!)] on J0 in 7207.82s, 1 test, 1 error <<< FAILURES!

[...truncated 2 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J0-20160803_100907_455.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps/java_pid7600.hprof ...
   [junit4] Heap dump file created [594472409 bytes in 1.938 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 11048 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:715: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid7600.hprof
* java_pid7602.hprof

Total time: 339 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1086 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1086/

18 tests failed.
FAILED:  org.apache.lucene.index.TestIndexSorting.testRandom3

Error Message:
this IndexWriter is closed

Stack Trace:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:740)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:754)
	at org.apache.lucene.index.IndexWriter.deleteDocuments(IndexWriter.java:1487)
	at org.apache.lucene.index.TestIndexSorting.testRandom3(TestIndexSorting.java:1271)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.lucene.util.packed.Packed64.<init>(Packed64.java:73)
	at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:972)
	at org.apache.lucene.util.packed.PackedInts.getMutable(PackedInts.java:939)
	at org.apache.lucene.util.packed.GrowableWriter.ensureCapacity(GrowableWriter.java:80)
	at org.apache.lucene.util.packed.GrowableWriter.set(GrowableWriter.java:88)
	at org.apache.lucene.util.packed.AbstractPagedMutable.set(AbstractPagedMutable.java:98)
	at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:152)
	at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:214)
	at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:310)
	at org.apache.lucene.util.fst.Builder.add(Builder.java:414)
	at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.writeFST(MemoryDocValuesConsumer.java:367)
	at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addSortedField(MemoryDocValuesConsumer.java:404)
	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addSortedField(PerFieldDocValuesFormat.java:119)
	at org.apache.lucene.codecs.DocValuesConsumer.mergeSortedField(DocValuesConsumer.java:641)
	at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:204)
	at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:167)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:111)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4287)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3864)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD:60A0CC82F3C90305]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:309)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:127)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=8177, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsRestartWhileUpdatingTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=8177, name=StoppableIndexingThread, state=RUNNABLE, group=TGRP-HdfsRestartWhileUpdatingTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)
	at java.util.HashMap.resize(HashMap.java:703)
	at java.util.HashMap.putVal(HashMap.java:628)
	at java.util.HashMap.put(HashMap.java:611)
	at java.util.HashSet.add(HashSet.java:219)
	at sun.security.ssl.SSLAlgorithmDecomposer.decomposes(SSLAlgorithmDecomposer.java:102)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:212)
	at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:243)
	at sun.security.util.AbstractAlgorithmConstraints.checkAlgorithm(AbstractAlgorithmConstraints.java:104)
	at sun.security.util.DisabledAlgorithmConstraints.permits(DisabledAlgorithmConstraints.java:91)
	at sun.security.ssl.SSLAlgorithmConstraints.permits(SSLAlgorithmConstraints.java:144)
	at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:349)
	at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:304)
	at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:626)
	at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:567)
	at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:110)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:363)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:40616/vr_g: Error CREATEing SolrCore 'test_unload_shard_and_collection_1': Unable to create core [test_unload_shard_and_collection_1] Caused by: Direct buffer memory

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40616/vr_g: Error CREATEing SolrCore 'test_unload_shard_and_collection_1': Unable to create core [test_unload_shard_and_collection_1] Caused by: Direct buffer memory
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:129)
	at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup

Error Message:


Stack Trace:
java.lang.AssertionError
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:112)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:     1) Thread[id=13065, name=qtp638794480-13065, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=13064, name=qtp1842265604-13064, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup: 
   1) Thread[id=13065, name=qtp638794480-13065, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=13064, name=qtp1842265604-13064, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.DocValuesMissingTest

Error Message:


Stack Trace:
java.lang.AssertionError
	at org.apache.lucene.util.TestRuleRestoreSystemProperties.before(TestRuleRestoreSystemProperties.java:52)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.DocValuesMissingTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.schema.DocValuesMissingTest:     1) Thread[id=13076, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-EventThread, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    2) Thread[id=13075, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-SendThread(127.0.0.1:42924), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.schema.DocValuesMissingTest: 
   1) Thread[id=13076, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-EventThread, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   2) Thread[id=13075, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-SendThread(127.0.0.1:42924), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.DocValuesMissingTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=13075, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-SendThread(127.0.0.1:42924), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.StringBuilder.<init>(StringBuilder.java:89)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1045)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=13075, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-SendThread(127.0.0.1:42924), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.StringBuilder.<init>(StringBuilder.java:89)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1045)
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:44123

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:44123
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:438)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=171445, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=171425, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=171420, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=171465, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=171440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=171430, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=171450, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=171455, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=171435, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=171460, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=171445, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=171425, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=171420, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=171465, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=171440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=171430, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=171450, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=171455, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=171435, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=171460, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=26091, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=22751, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=24577, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=21670, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=23026, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=23437, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=23196, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=28352, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=22749, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=28355, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=21939, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=24578, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=23025, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=21669, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=23440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=28354, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=23027, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=28828, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=23198, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=22082, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=28830, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=26090, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=26089, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=23024, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=26088, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=22084, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=22316, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=28353, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=22317, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=23436, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=21940, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=22752, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=22318, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=22319, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=23197, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=22750, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=22081, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=21942, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=24579, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=28827, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=24576, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=23199, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=23438, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=28829, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=21941, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=22083, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=21666, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=21665, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=26091, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=22751, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=24577, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=21670, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=23026, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=23437, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=23196, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=28352, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=22749, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=28355, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=21939, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=24578, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=23025, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=21669, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=23440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=28354, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=23027, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=28828, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=23198, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=22082, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=28830, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=26090, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=26089, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=23024, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=26088, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=22084, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=22316, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=28353, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=22317, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=23436, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=21940, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=22752, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=22318, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=22319, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=23197, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=22750, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=22081, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=21942, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=24579, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=28827, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=24576, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=23199, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=23438, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=28829, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=21941, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=22083, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=21666, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=21665, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=73916, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=74282, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=74284, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=74283, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=73917, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=73915, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=74281, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=73914, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=73916, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=74282, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=74284, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=74283, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=73917, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=73915, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=74281, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=73914, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! 

[...truncated too long message...]

, waiting 100 ms before checking again.
   [junit4]   2> 7648000 INFO  (qtp1842265604-11323) [n:127.0.0.1:36102__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard7_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7648542 WARN  (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@7cd5d022) [    ] o.a.h.h.s.n.NameNodeResourceChecker Space available on volume '/dev/sdb1' is 0, which is below the configured reserved amount 104857600
   [junit4]   2> 7648542 WARN  (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@7cd5d022) [    ] o.a.h.h.s.n.FSNamesystem NameNode low on available disk space. Already in safe mode.
   [junit4]   2> 7648549 INFO  (qtp1842265604-11323) [n:127.0.0.1:36102__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard7_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7648549 INFO  (qtp1587451453-8299) [n:127.0.0.1:41500__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard5_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7649064 INFO  (qtp1587451453-8299) [n:127.0.0.1:41500__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard5_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7649064 INFO  (qtp1842265604-11323) [n:127.0.0.1:36102__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard7_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7649064 WARN  (zkCallback-451-thread-15-processing-n:127.0.0.1:38967__hu-EventThread) [n:127.0.0.1:38967__hu    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6c5a9742 name:ZooKeeperConnection Watcher:127.0.0.1:42924/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 7649064 WARN  (zkCallback-451-thread-15-processing-n:127.0.0.1:38967__hu-EventThread) [n:127.0.0.1:38967__hu    ] o.a.s.c.c.ConnectionManager zkClient has disconnected
   [junit4]   2> 7649593 INFO  (qtp1842265604-11323) [n:127.0.0.1:36102__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard7_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7649593 INFO  (qtp1587451453-8299) [n:127.0.0.1:41500__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard5_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7650145 INFO  (qtp1587451453-8299) [n:127.0.0.1:41500__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard5_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> 7650145 INFO  (qtp1842265604-11323) [n:127.0.0.1:36102__hu    ] o.a.s.c.SolrCore Core awholynewstresscollection_collection0_0_shard7_replica3 is not yet closed, waiting 100 ms before checking again.
   [junit4]   2> Aug 02, 2016 1:16:24 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=13075, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-SendThread(127.0.0.1:42924), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]   2>         at java.lang.StringBuilder.<init>(StringBuilder.java:89)
   [junit4]   2>         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1045)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {other_tl1=FSTOrd50, range_facet_l_dv=BlockTreeOrds(blocksize=128), _version_=FSTOrd50, multiDefault=PostingsFormat(name=MockRandom), a_t=FSTOrd50, intDefault=FSTOrd50, a_i1=FSTOrd50, id=BlockTreeOrds(blocksize=128), range_facet_i_dv=FSTOrd50, text=PostingsFormat(name=Memory doPackFST= false), range_facet_l=FSTOrd50, timestamp=FSTOrd50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1238, maxMBSortInHeap=6.764167308203172, sim=ClassicSimilarity, locale=zh-HK, timezone=Asia/Jakarta
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=12387952,total=509083648
   [junit4]   2> NOTE: All tests run in this JVM: [TestHashQParserPlugin, ZkControllerTest, SmileWriterTest, CdcrUpdateLogTest, JavabinLoaderTest, TestSubQueryTransformer, TestFiltering, SolrIndexConfigTest, TestCollapseQParserPlugin, HdfsRecoveryZkTest, TestManagedResource, TestTolerantUpdateProcessorCloud, DocumentBuilderTest, DeleteReplicaTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestOrdValues, TestDistribIDF, DistributedIntervalFacetingTest, TestSchemaSimilarityResource, SolrIndexSplitterTest, TestDistributedGrouping, TestQuerySenderNoQuery, AsyncMigrateRouteKeyTest, DistributedMLTComponentTest, TestUseDocValuesAsStored2, TestDefaultSearchFieldResource, TestTrie, TestBinaryResponseWriter, ReplicationFactorTest, DirectSolrSpellCheckerTest, TestManagedStopFilterFactory, TestLRUStatsCache, TestTestInjection, TestSolrQueryResponse, DocValuesTest, OverseerTaskQueueTest, HdfsCollectionsAPIDistributedZkTest, TestReplicationHandlerBackup, DocValuesMissingTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DocValuesMissingTest -Dtests.seed=E8F4F3585D356EFD -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh-HK -Dtests.timezone=Asia/Jakarta -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4]   2> 7650149 INFO  (zkCallback-451-thread-15-processing-n:127.0.0.1:38967__hu-EventThread) [n:127.0.0.1:38967__hu    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6c5a9742 name:ZooKeeperConnection Watcher:127.0.0.1:42924/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4] ERROR   0.00s J1 | DocValuesMissingTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.schema.DocValuesMissingTest: 
   [junit4]    >    1) Thread[id=13076, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-EventThread, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >    2) Thread[id=13075, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-SendThread(127.0.0.1:42924), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   [junit4]    >         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=13075, name=zkCallback-458-thread-13-processing-n:127.0.0.1:37964__hu-SendThread(127.0.0.1:42924), state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.StringBuilder.<init>(StringBuilder.java:89)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1045)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E8F4F3585D356EFD]:0)
   [junit4] Completed [561/625 (11!)] on J1 in 38.22s, 0 tests, 1 failure, 2 errors <<< FAILURES!

[...truncated 193 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160802_040851_444.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps/java_pid17934.hprof ...
   [junit4] Heap dump file created [730202302 bytes in 8.617 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 11049 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:715: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid17934.hprof
* java_pid17946.hprof
* java_pid19768.hprof

Total time: 264 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1085 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1085/

11 tests failed.
FAILED:  org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit

Error Message:
this IndexWriter is closed

Stack Trace:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:740)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:754)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3145)
	at org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:228)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=25103, name=collection5, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25103, name=collection5, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:38274/fl_qq: Could not find collection : awholynewstresscollection_collection5_4
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:413)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:366)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1291)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:995)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest:     1) Thread[id=91126, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=91121, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=91136, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=91156, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=91116, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=91146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=91161, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=91131, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=91151, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=91141, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest: 
   1) Thread[id=91126, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=91121, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=91136, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=91156, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=91116, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=91146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=91161, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=91131, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=91151, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=91141, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ConcurrentDeleteAndCreateCollectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:58222/zur/h

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:58222/zur/h
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:619)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:513)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=8441, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=5254, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=1285, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=3659, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=875, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=4091, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=4843, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=8443, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=7099, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=5017, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=5256, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=876, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=7101, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=1526, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=1145, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=8442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=7100, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=4568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=5016, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=1524, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=4842, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=4566, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=4094, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=3658, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=4092, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=3657, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=877, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=1146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=1523, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=7098, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=5257, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=5255, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=878, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=1284, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=1286, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=4093, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=5015, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=4840, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=4841, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=5014, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=1144, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=3656, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=1147, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=1525, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=4567, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=8440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=4565, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=1283, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=8441, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=5254, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=1285, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=3659, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=875, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=4091, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=4843, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=8443, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=7099, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=5017, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=5256, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=876, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  13) Thread[id=7101, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  14) Thread[id=1526, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  15) Thread[id=1145, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  16) Thread[id=8442, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  17) Thread[id=7100, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  18) Thread[id=4568, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  19) Thread[id=5016, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  20) Thread[id=1524, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  21) Thread[id=4842, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  22) Thread[id=4566, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  23) Thread[id=4094, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  24) Thread[id=3658, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  25) Thread[id=4092, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  26) Thread[id=3657, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  27) Thread[id=877, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  28) Thread[id=1146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  29) Thread[id=1523, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  30) Thread[id=7098, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  31) Thread[id=5257, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  32) Thread[id=5255, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  33) Thread[id=878, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  34) Thread[id=1284, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  35) Thread[id=1286, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  36) Thread[id=4093, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  37) Thread[id=5015, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  38) Thread[id=4840, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  39) Thread[id=4841, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  40) Thread[id=5014, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  41) Thread[id=1144, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  42) Thread[id=3656, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  43) Thread[id=1147, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  44) Thread[id=1525, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  45) Thread[id=4567, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  46) Thread[id=8440, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  47) Thread[id=4565, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  48) Thread[id=1283, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:     1) Thread[id=9996, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=9997, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=9999, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=10608, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=10607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=10609, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=9998, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=10610, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 
   1) Thread[id=9996, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=9997, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=9999, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=10608, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=10607, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=10609, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=9998, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=10610, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient]
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
	at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=100036, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=102310, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=100294, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=100810, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=100037, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=102309, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=100811, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=100295, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=102193, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=102194, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=100551, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=100550, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest: 
   1) Thread[id=100036, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=102310, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=100294, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=100810, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=100037, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=102309, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=100811, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=100295, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=102193, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=102194, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=100551, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=100550, name=Connection evictor, state=TIMED_WAITING, group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4B2A12E979929B20]:0)




Build Log:
Started by timer

[EnvInject] - Loading node environment variables.

Building remotely on lucene in workspace /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master

 > git rev-parse --is-inside-work-tree # timeout=10

Fetching changes from the remote Git repository

 > git config remote.origin.url https://git-wip-us.apache.org/repos/asf/lucene-solr.git # timeout=10

Cleaning workspace

 > git rev-parse --verify HEAD # timeout=10

Resetting working tree

 > git reset --hard # timeout=10

 > git clean -fdx # timeout=10

Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/lucene-solr.git

 > git --version # timeout=10

 > git -c core.askpass=true fetch --tags --progress https://git-wip-us.apache.org/repos/asf/lucene-solr.git +refs/heads/*:refs/remotes/origin/*

 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10

 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10

Checking out Revision d86c369533f0d3fc3cda9d15440521f39d84b750 (refs/remotes/origin/master)

 > git config core.sparsecheckout # timeout=10

 > git checkout -f d86c369533f0d3fc3cda9d15440521f39d84b750

 > git rev-list 53a34b312e78ce6f56c0bb41304ac834b28b9534 # timeout=10

No emails were triggered.

[Lucene-Solr-NightlyTests-master] $ /home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/bin/ant -Dtests.multiplier=2 -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt jenkins-nightly

Buildfile: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml


jenkins-nightly:


-print-java-info:

[java-info] java version "1.8.0_74"

[java-info] Java(TM) SE Runtime Environment (1.8.0_74-b02, Oracle Corporation)

[java-info] Java HotSpot(TM) 64-Bit Server VM (25.74-b02, Oracle Corporation)

[java-info] Test args: []


clean:


clean:


clean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: Apache Ivy 2.3.0 - 20130110142753 :: http://ant.apache.org/ivy/ ::

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve-groovy:

[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working

[ivy:cachepath] 	confs: [default]

[ivy:cachepath] 	found org.codehaus.groovy#groovy-all;2.4.6 in public

[ivy:cachepath] :: resolution report :: resolve 91ms :: artifacts dl 2ms

	---------------------------------------------------------------------

	|                  |            modules            ||   artifacts   |

	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|

	---------------------------------------------------------------------

	|      default     |   1   |   0   |   0   |   0   ||   1   |   0   |

	---------------------------------------------------------------------


-test-with-heapdumps-enabled:

     [echo] Java HotSpot(TM) 64-Bit Server VM: Enabling heap dumps on OutOfMemoryError to dir '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps'.

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve-groovy:


test:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


-clover.load:


resolve-groovy:


-init-totals:


test-core:


-clover.disable:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


-clover.load:


-clover.classpath:


-clover.setup:


clover:


-check-git-state:


-git-cleanroot:


-copy-git-state:

     [copy] Copying 1 file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build


git-autoclean:


resolve:


init:


compile-core:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java

    [javac] Compiling 779 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java

    [javac] Note: Some input files use or override a deprecated API.

    [javac] Note: Recompile with -Xlint:deprecation for details.

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/document/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/bkd/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/compressing/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene60/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/index/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/fst/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/tokenattributes/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/search/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene62/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/packed/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/mutable/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/perfield/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/analysis/standard/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/store/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/blocktree/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/search/spans/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene54/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/search/similarities/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/util/automaton/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene53/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/codecs/lucene50/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java/org/apache/lucene/geo/package-info.class

     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/core/classes/java


compile-test-framework:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


init:


compile-lucene-core:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


init:


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


compile-core:


compile-codecs:


-check-git-state:


-git-cleanroot:


-copy-git-state:


git-autoclean:


ivy-availability-check:


ivy-fail:


ivy-configure:

[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/top-level-ivy-settings.xml


resolve:


common.init:


compile-lucene-core:


init:


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


compile-core:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java

    [javac] Compiling 66 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/bloom/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/memory/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/simpletext/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/blockterms/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java/org/apache/lucene/codecs/blocktreeords/package-info.class

     [copy] Copying 3 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/codecs/classes/java


-clover.disable:


-clover.load:


-clover.classpath:


-clover.setup:


clover:


common.compile-core:

    [mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java

    [javac] Compiling 185 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java

    [javac] Note: Some input files use or override a deprecated API.

    [javac] Note: Recompile with -Xlint:deprecation for details.

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/asserting/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/ramonly/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/mockfile/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/cheapbastard/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/mockrandom/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/cranky/package-info.class

    [javac] Creating empty /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/test-framework/classes/java/org/apache/lucene/codecs/compressing/dummy/package-info.class

     [copy] 

[...truncated too long message...]

r/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.7.2-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.7.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-core-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-mapper-asl-1.9.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/netty-all-4.0.36.Final.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160801_043458_219.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J2-20160801_043458_219.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:771: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:707: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:536: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1443: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:984: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 244 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
ERROR: Failed to archive {README.txt=README.txt, heapdumps/java_pid17575.hprof=heapdumps/java_pid17575.hprof, solr/build/solr-core/test/temp/junit4-J1-20160801_043458_218.events=solr/build/solr-core/test/temp/junit4-J1-20160801_043458_218.events, heapdumps/java_pid14871.hprof=heapdumps/java_pid14871.hprof, heapdumps/java_pid17570.hprof=heapdumps/java_pid17570.hprof, solr/build/solr-core/test/temp/junit4-J2-20160801_043458_219.events=solr/build/solr-core/test/temp/junit4-J2-20160801_043458_219.events, solr/build/solr-core/test/temp/junit4-J0-20160801_043458_218.events=solr/build/solr-core/test/temp/junit4-J0-20160801_043458_218.events} due to internal error; falling back to full archiving
java.lang.NullPointerException
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any