You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/03/06 13:30:33 UTC

[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 707 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/707/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, MDCAwareThreadPoolExecutor, MMapDirectory, MMapDirectory, SolrCore] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:348)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:677)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:894)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)  at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)  at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:894)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)  at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:479)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:888)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)  at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:984)  at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)  at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, MDCAwareThreadPoolExecutor, MMapDirectory, MMapDirectory, SolrCore]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:348)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:677)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:894)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:894)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:479)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:888)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:984)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


	at __randomizedtesting.SeedInfo.seed([647878CC0446E339]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:302)
	at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores:     1) Thread[id=10439, name=searcherExecutor-4386-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 
   1) Thread[id=10439, name=searcherExecutor-4386-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([647878CC0446E339]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=10439, name=searcherExecutor-4386-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=10439, name=searcherExecutor-4386-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([647878CC0446E339]:0)




Build Log:
[...truncated 11906 lines...]
   [junit4] Suite: org.apache.solr.core.TestLazyCores
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/init-core-data-001
   [junit4]   2> 1298362 INFO  (SUITE-TestLazyCores-seed#[647878CC0446E339]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1298365 INFO  (SUITE-TestLazyCores-seed#[647878CC0446E339]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1298367 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLazySearch
   [junit4]   2> 1298401 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 1298402 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    ] o.a.s.c.SolrCores Allocating transient cache for 4 transient cores
   [junit4]   2> 1298467 WARN  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1298483 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1298483 WARN  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1298485 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1298488 WARN  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1298489 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1298492 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.s.IndexSchema [collection2] Schema name=tiny
   [junit4]   2> 1298493 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.s.IndexSchema [collection5] Schema name=tiny
   [junit4]   2> 1298495 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1298495 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1298498 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.c.CoreContainer Creating SolrCore 'collection5' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection5
   [junit4]   2> 1298498 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection2
   [junit4]   2> 1298499 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=tiny
   [junit4]   2> 1298499 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection2/data/]
   [junit4]   2> 1298499 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.c.SolrCore [[collection5] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection5], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection5/data/]
   [junit4]   2> 1298501 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1298501 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1298502 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1298504 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection1
   [junit4]   2> 1298505 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection1/data/]
   [junit4]   2> 1298508 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1298621 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1298621 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1298621 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1298621 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1298622 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1298623 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@59fe5921[collection2] main]
   [junit4]   2> 1298623 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection2/conf
   [junit4]   2> 1298623 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1298624 INFO  (searcherExecutor-4170-thread-1-processing-x:collection2) [    x:collection2] o.a.s.c.SolrCore [collection2] Registered new searcher Searcher@59fe5921[collection2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1298624 INFO  (coreLoadExecutor-4168-thread-2) [    x:collection2] o.a.s.c.SolrCores Opening transient core collection2
   [junit4]   2> 1298627 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1298627 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.s.SolrIndexSearcher Opening [Searcher@2a5f4f0e[collection5] main]
   [junit4]   2> 1298628 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection5/conf
   [junit4]   2> 1298628 INFO  (coreLoadExecutor-4168-thread-3) [    x:collection5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1298628 INFO  (searcherExecutor-4169-thread-1-processing-x:collection5) [    x:collection5] o.a.s.c.SolrCore [collection5] Registered new searcher Searcher@2a5f4f0e[collection5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1298631 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1298631 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1298632 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1298633 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c4db8e5[collection1] main]
   [junit4]   2> 1298633 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection1/conf
   [junit4]   2> 1298634 INFO  (coreLoadExecutor-4168-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1298634 INFO  (searcherExecutor-4171-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3c4db8e5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1298650 WARN  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1298651 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1298659 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema [collection4] Schema name=tiny
   [junit4]   2> 1298663 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1298666 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating SolrCore 'collection4' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection4
   [junit4]   2> 1298666 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCore [[collection4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection4/data/]
   [junit4]   2> 1298669 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=104868399, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1298756 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1298756 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1298757 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=49, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6129598777955085]
   [junit4]   2> 1298758 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f6c9144[collection4] main]
   [junit4]   2> 1298758 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-001/collection4/conf
   [junit4]   2> 1298759 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1298759 INFO  (searcherExecutor-4181-thread-1-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [collection4] Registered new searcher Searcher@2f6c9144[collection4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1298760 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1298760 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2bbcb069
   [junit4]   2> 1298779 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@13260b3c[collection4] main]
   [junit4]   2> 1298779 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1298779 INFO  (searcherExecutor-4181-thread-1-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [collection4] Registered new searcher Searcher@13260b3c[collection4] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):c7)))}
   [junit4]   2> 1298780 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.S.Request [collection4]  webapp=null path=null params={q={!prefix+f%3Dv_t}hel&wt=xml} hits=2 status=0 QTime=0
   [junit4]   2> 1298785 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.S.Request [collection4]  webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=2 status=0 QTime=0
   [junit4]   2> 1298785 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.S.Request [collection4]  webapp=null path=null params={q={!raw+f%3Dv_t}Hello&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1298786 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.S.Request [collection4]  webapp=null path=null params={q={!raw+f%3Dv_f}1.5&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1298786 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=0 status=0 QTime=0
   [junit4]   2> 1298786 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1537263990
   [junit4]   2> 1298790 INFO  (coreCloseExecutor-4185-thread-1-processing-x:collection4) [    x:collection5] o.a.s.c.SolrCore [collection5]  CLOSING SolrCore org.apache.solr.core.SolrCore@17ef1517
   [junit4]   2> 1298790 INFO  (coreCloseExecutor-4185-thread-2-processing-x:collection4) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1ab0a03b
   [junit4]   2> 1298806 INFO  (coreCloseExecutor-4185-thread-4-processing-x:collection4) [    x:collection2] o.a.s.c.SolrCore [collection2]  CLOSING SolrCore org.apache.solr.core.SolrCore@4580a38c
   [junit4]   2> 1298809 INFO  (coreCloseExecutor-4185-thread-3-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [collection4]  CLOSING SolrCore org.apache.solr.core.SolrCore@17de8454
   [junit4]   2> 1298815 INFO  (coreCloseExecutor-4185-thread-3-processing-x:collection4) [    x:collection4] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection4
   [junit4]   2> 1298815 INFO  (coreCloseExecutor-4185-thread-4-processing-x:collection4) [    x:collection2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection2
   [junit4]   2> 1298815 INFO  (coreCloseExecutor-4185-thread-1-processing-x:collection4) [    x:collection5] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection5
   [junit4]   2> 1298815 INFO  (coreCloseExecutor-4185-thread-2-processing-x:collection4) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1298816 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1298816 INFO  (TEST-TestLazyCores.testLazySearch-seed#[647878CC0446E339]) [    x:collection4] o.a.s.SolrTestCaseJ4 ###Ending testLazySearch
   [junit4]   2> 1298818 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.SolrTestCaseJ4 ###Starting testCreateSame
   [junit4]   2> 1298848 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 1298849 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCores Allocating transient cache for 4 transient cores
   [junit4]   2> 1298933 WARN  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1298933 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1298941 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=tiny
   [junit4]   2> 1298945 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1298947 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection1
   [junit4]   2> 1298948 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection1/data/]
   [junit4]   2> 1298950 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1299000 WARN  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299001 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1299022 WARN  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299023 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1299032 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.s.IndexSchema [collection5] Schema name=tiny
   [junit4]   2> 1299035 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.s.IndexSchema [collection2] Schema name=tiny
   [junit4]   2> 1299037 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1299040 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection2
   [junit4]   2> 1299040 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection2/data/]
   [junit4]   2> 1299042 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1299053 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1299053 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1299054 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1299054 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3d95ff75[collection1] main]
   [junit4]   2> 1299055 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection1/conf
   [junit4]   2> 1299055 INFO  (coreLoadExecutor-4192-thread-1-processing-x:collection4) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1299056 INFO  (searcherExecutor-4193-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3d95ff75[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1299099 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1299099 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1299100 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1299100 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1299102 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.CoreContainer Creating SolrCore 'collection5' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection5
   [junit4]   2> 1299102 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.SolrCore [[collection5] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection5], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection5/data/]
   [junit4]   2> 1299104 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1299106 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@508751b2[collection2] main]
   [junit4]   2> 1299106 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection2/conf
   [junit4]   2> 1299107 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1299107 INFO  (searcherExecutor-4194-thread-1-processing-x:collection2) [    x:collection2] o.a.s.c.SolrCore [collection2] Registered new searcher Searcher@508751b2[collection2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1299107 INFO  (coreLoadExecutor-4192-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.SolrCores Opening transient core collection2
   [junit4]   2> 1299179 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1299179 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1299180 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1299180 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ef8191f[collection5] main]
   [junit4]   2> 1299181 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection5/conf
   [junit4]   2> 1299181 INFO  (coreLoadExecutor-4192-thread-3-processing-x:collection4) [    x:collection5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1299182 INFO  (searcherExecutor-4201-thread-1-processing-x:collection5) [    x:collection5] o.a.s.c.SolrCore [collection5] Registered new searcher Searcher@5ef8191f[collection5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1299197 WARN  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299197 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1299205 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema [collection4] Schema name=tiny
   [junit4]   2> 1299209 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1299211 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating SolrCore 'collection4' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection4
   [junit4]   2> 1299212 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCore [[collection4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection4/data/]
   [junit4]   2> 1299214 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=71.26171875, floorSegmentMB=0.9052734375, forceMergeDeletesPctAllowed=21.64072590646632, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4817249187360563
   [junit4]   2> 1299263 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1299263 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1299264 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=22.4248046875, floorSegmentMB=0.8671875, forceMergeDeletesPctAllowed=7.37524458096741, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.48125648752076244
   [junit4]   2> 1299264 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@6042910c[collection4] main]
   [junit4]   2> 1299265 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection4/conf
   [junit4]   2> 1299266 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1299266 INFO  (searcherExecutor-4206-thread-1-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [collection4] Registered new searcher Searcher@6042910c[collection4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1299283 WARN  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299284 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1299292 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema [collection6] Schema name=tiny
   [junit4]   2> 1299295 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1299298 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating SolrCore 'collection6' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection6
   [junit4]   2> 1299299 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCore [[collection6] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection6], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection6/data/]
   [junit4]   2> 1299301 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.711374065015999]
   [junit4]   2> 1299400 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1299400 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1299401 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=7.158203125, floorSegmentMB=1.20703125, forceMergeDeletesPctAllowed=9.250244964588955, segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.48601221452907706
   [junit4]   2> 1299403 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@498e2b13[collection6] main]
   [junit4]   2> 1299403 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-002/collection6/conf
   [junit4]   2> 1299404 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1299404 INFO  (searcherExecutor-4210-thread-1-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [collection6] Registered new searcher Searcher@498e2b13[collection6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1299404 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCores Opening transient core collection6
   [junit4]   2> 1299420 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command schema=schema.xml&dataDir=t12&name=collection2&action=CREATE&config=solrconfig.xml&wt=xml
   [junit4]   2> 1299420 WARN  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating a core with existing name is not allowed
   [junit4]   2> 1299420 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command schema=schema.xml&dataDir=t14&name=collection4&action=CREATE&config=solrconfig.xml&wt=xml
   [junit4]   2> 1299421 WARN  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating a core with existing name is not allowed
   [junit4]   2> 1299421 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command schema=schema.xml&dataDir=t15&name=collection5&action=CREATE&config=solrconfig.xml&wt=xml
   [junit4]   2> 1299421 WARN  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating a core with existing name is not allowed
   [junit4]   2> 1299421 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command schema=schema.xml&dataDir=t16&name=collection6&action=CREATE&config=solrconfig.xml&wt=xml
   [junit4]   2> 1299422 WARN  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating a core with existing name is not allowed
   [junit4]   2> 1299422 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Shutting down CoreContainer instance=625928939
   [junit4]   2> 1299422 INFO  (coreCloseExecutor-4214-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@353562a1
   [junit4]   2> 1299438 INFO  (coreCloseExecutor-4214-thread-3-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [collection4]  CLOSING SolrCore org.apache.solr.core.SolrCore@67cf4c48
   [junit4]   2> 1299438 INFO  (coreCloseExecutor-4214-thread-5-processing-x:collection4) [    x:collection6] o.a.s.c.SolrCore [collection6]  CLOSING SolrCore org.apache.solr.core.SolrCore@7c30e867
   [junit4]   2> 1299439 INFO  (coreCloseExecutor-4214-thread-4-processing-x:collection4) [    x:collection2] o.a.s.c.SolrCore [collection2]  CLOSING SolrCore org.apache.solr.core.SolrCore@2bac02f3
   [junit4]   2> 1299454 INFO  (coreCloseExecutor-4214-thread-2-processing-x:collection4) [    x:collection5] o.a.s.c.SolrCore [collection5]  CLOSING SolrCore org.apache.solr.core.SolrCore@7e30fa1
   [junit4]   2> 1299472 INFO  (coreCloseExecutor-4214-thread-1-processing-x:collection4) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1299474 INFO  (coreCloseExecutor-4214-thread-3-processing-x:collection4) [    x:collection4] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection4
   [junit4]   2> 1299474 INFO  (coreCloseExecutor-4214-thread-2-processing-x:collection4) [    x:collection5] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection5
   [junit4]   2> 1299475 INFO  (coreCloseExecutor-4214-thread-4-processing-x:collection4) [    x:collection2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection2
   [junit4]   2> 1299476 INFO  (coreCloseExecutor-4214-thread-5-processing-x:collection4) [    x:collection6] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection6
   [junit4]   2> 1299477 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1299477 INFO  (TEST-TestLazyCores.testCreateSame-seed#[647878CC0446E339]) [    x:collection4] o.a.s.SolrTestCaseJ4 ###Ending testCreateSame
   [junit4]   2> 1299479 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.SolrTestCaseJ4 ###Starting testCreateTransientFromAdmin
   [junit4]   2> 1299719 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 1299724 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCores Allocating transient cache for 4 transient cores
   [junit4]   2> 1299781 WARN  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299781 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1299782 WARN  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299786 WARN  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1299786 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1299787 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1299792 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.s.IndexSchema [collection2] Schema name=tiny
   [junit4]   2> 1299792 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=tiny
   [junit4]   2> 1299796 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1299797 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection1
   [junit4]   2> 1299798 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection1/data/]
   [junit4]   2> 1299799 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1299803 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.s.IndexSchema [collection5] Schema name=tiny
   [junit4]   2> 1299805 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1299806 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.CoreContainer Creating SolrCore 'collection5' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection5
   [junit4]   2> 1299807 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.c.SolrCore [[collection5] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection5], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection5/data/]
   [junit4]   2> 1299808 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1299812 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1299813 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection2
   [junit4]   2> 1299814 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection2/data/]
   [junit4]   2> 1299815 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5815631727240332]
   [junit4]   2> 1303034 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1303034 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1303034 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1303034 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1303034 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1303034 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1303035 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1303035 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1303035 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=86.4375, floorSegmentMB=0.294921875, forceMergeDeletesPctAllowed=2.3413361893617877, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1303037 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@58cd671b[collection2] main]
   [junit4]   2> 1303037 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.s.SolrIndexSearcher Opening [Searcher@1db7789e[collection5] main]
   [junit4]   2> 1303037 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@417ccf91[collection1] main]
   [junit4]   2> 1303037 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection2/conf
   [junit4]   2> 1303037 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection1/conf
   [junit4]   2> 1303037 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/collection5/conf
   [junit4]   2> 1303038 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1303038 INFO  (coreLoadExecutor-4221-thread-3-processing-x:collection4) [    x:collection5] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1303038 INFO  (searcherExecutor-4224-thread-1-processing-x:collection2) [    x:collection2] o.a.s.c.SolrCore [collection2] Registered new searcher Searcher@58cd671b[collection2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1303038 INFO  (coreLoadExecutor-4221-thread-1-processing-x:collection4) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1303038 INFO  (searcherExecutor-4223-thread-1-processing-x:collection5) [    x:collection5] o.a.s.c.SolrCore [collection5] Registered new searcher Searcher@1db7789e[collection5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1303038 INFO  (coreLoadExecutor-4221-thread-2-processing-x:collection4) [    x:collection2] o.a.s.c.SolrCores Opening transient core collection2
   [junit4]   2> 1303038 INFO  (searcherExecutor-4222-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@417ccf91[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1303055 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command loadOnStartup=true&transient=true&name=core1&action=CREATE&wt=xml
   [junit4]   2> 1303067 WARN  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1303068 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1303077 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema [core1] Schema name=tiny
   [junit4]   2> 1303080 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1303082 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating SolrCore 'core1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core1
   [junit4]   2> 1303083 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCore [[core1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core1/data/]
   [junit4]   2> 1303086 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=23.873046875, floorSegmentMB=0.7119140625, forceMergeDeletesPctAllowed=21.611048391475702, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35872941329914776
   [junit4]   2> 1304805 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1304805 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1304806 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=68.3857421875, floorSegmentMB=0.517578125, forceMergeDeletesPctAllowed=0.32420685754323886, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1278553159671481
   [junit4]   2> 1304807 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@3dca3911[core1] main]
   [junit4]   2> 1304807 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core1/conf
   [junit4]   2> 1304808 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1304808 INFO  (searcherExecutor-4235-thread-1-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@3dca3911[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1304808 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCores Opening transient core core1
   [junit4]   2> 1304809 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command loadOnStartup=false&transient=true&name=core2&action=CREATE&wt=xml
   [junit4]   2> 1304823 WARN  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1304824 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1304831 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema [core2] Schema name=tiny
   [junit4]   2> 1304835 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1304838 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating SolrCore 'core2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core2
   [junit4]   2> 1304838 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCore [[core2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core2/data/]
   [junit4]   2> 1304840 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=81.787109375, floorSegmentMB=1.6796875, forceMergeDeletesPctAllowed=19.568945951567283, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3165037767344521
   [junit4]   2> 1304915 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1304915 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1304916 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=52.6806640625, floorSegmentMB=1.2568359375, forceMergeDeletesPctAllowed=4.763647887577749, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5336047796524767
   [junit4]   2> 1304916 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f69827b[core2] main]
   [junit4]   2> 1304917 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core2/conf
   [junit4]   2> 1304917 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1304917 INFO  (searcherExecutor-4240-thread-1-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [core2] Registered new searcher Searcher@2f69827b[core2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1304917 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCores Opening transient core core2
   [junit4]   2> 1304917 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command loadOnStartup=true&transient=true&name=core3&action=CREATE&wt=xml
   [junit4]   2> 1304927 WARN  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1304928 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1304935 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema [core3] Schema name=tiny
   [junit4]   2> 1304937 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1304939 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating SolrCore 'core3' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core3
   [junit4]   2> 1304940 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCore [[core3] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core3], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core3/data/]
   [junit4]   2> 1304943 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=31.7822265625, floorSegmentMB=1.421875, forceMergeDeletesPctAllowed=4.212322084347625, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8386079263321351
   [junit4]   2> 1305016 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1305016 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1305017 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1305017 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@404604dc[core3] main]
   [junit4]   2> 1305018 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core3/conf
   [junit4]   2> 1305018 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1305018 INFO  (searcherExecutor-4245-thread-1-processing-x:collection4) [    x:collection4] o.a.s.c.SolrCore [core3] Registered new searcher Searcher@404604dc[core3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1305018 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCores Opening transient core core3
   [junit4]   2> 1305018 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.h.a.CoreAdminOperation core create command loadOnStartup=false&transient=true&name=core4&action=CREATE&wt=xml
   [junit4]   2> 1305040 WARN  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1305041 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1305047 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema [core4] Schema name=tiny
   [junit4]   2> 1305050 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
   [junit4]   2> 1305051 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.CoreContainer Creating SolrCore 'core4' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core4
   [junit4]   2> 1305052 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.c.SolrCore [[core4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_647878CC0446E339-001/tempDir-003/core4/data/]
   [junit4]   2> 1305055 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=41.720703125, floorSegmentMB=0.2265625, forceMergeDeletesPctAllowed=8.321860534090142, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1305106 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1305106 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1305107 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=78.451171875, floorSegmentMB=1.2939453125, forceMergeDeletesPctAllowed=22.05063769056467, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1305107 INFO  (TEST-TestLazyCores.testCreateTransientFromAdmin-seed#[647878CC0446E339]) [    x:collection4] o.a.s.s.SolrIndexSearcher Opening [Searcher@64db215b[core4] main]
   [junit4]   2> 1305107 INFO  (TEST-TestLazyCores.testCreateTran

[...truncated too long message...]

 TestUseDocValuesAsStored2, TestBinaryResponseWriter, CloudExitableDirectoryReaderTest, HighlighterTest, TestBlendedInfixSuggestions, TestLFUCache, DirectoryFactoryTest, TestConfigsApi, HttpSolrCallGetCoreTest, DOMUtilTest, TestSolrQueryParserDefaultOperatorResource, TestLazyCores]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLazyCores -Dtests.seed=647878CC0446E339 -Dtests.slow=true -Dtests.locale=en-PH -Dtests.timezone=America/Yellowknife -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestLazyCores (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 5 object(s) that were not released!!! [MMapDirectory, MDCAwareThreadPoolExecutor, MMapDirectory, MMapDirectory, SolrCore]
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:348)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:677)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:894)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
   [junit4]    > 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
   [junit4]    > 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
   [junit4]    > 	at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:98)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:709)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:894)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
   [junit4]    > 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MMapDirectory
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:479)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:888)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
   [junit4]    > 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore
   [junit4]    > 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:984)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:811)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:917)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$3(CoreContainer.java:552)
   [junit4]    > 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([647878CC0446E339]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:302)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 
   [junit4]    >    1) Thread[id=10439, name=searcherExecutor-4386-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([647878CC0446E339]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=10439, name=searcherExecutor-4386-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([647878CC0446E339]:0)
   [junit4] Completed [325/697 (1!)] on J0 in 278.53s, 9 tests, 1 failure, 2 errors, 1 skipped <<< FAILURES!

[...truncated 63906 lines...]


[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 711 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/711/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew

Error Message:
expected:<200> but was:<403>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<403>
	at __randomizedtesting.SeedInfo.seed([4A2C8E19EED5E8D9:7DB77A07D619357D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.renewDelegationToken(TestDelegationWithHadoopAuth.java:118)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.verifyDelegationTokenRenew(TestDelegationWithHadoopAuth.java:302)
	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew(TestDelegationWithHadoopAuth.java:319)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10904 lines...]
   [junit4] Suite: org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_4A2C8E19EED5E8D9-001/init-core-data-001
   [junit4]   2> 19715 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[4A2C8E19EED5E8D9]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 19717 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[4A2C8E19EED5E8D9]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 19719 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[4A2C8E19EED5E8D9]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_4A2C8E19EED5E8D9-001/tempDir-001
   [junit4]   2> 19729 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[4A2C8E19EED5E8D9]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 19732 INFO  (Thread-17) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 19732 INFO  (Thread-17) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 19844 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[4A2C8E19EED5E8D9]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:47992
   [junit4]   2> 20035 INFO  (jetty-launcher-15-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 20037 INFO  (jetty-launcher-15-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@359dc0e8{/solr,null,AVAILABLE}
   [junit4]   2> 20040 INFO  (jetty-launcher-15-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5d9200c4{HTTP/1.1,[http/1.1]}{127.0.0.1:36864}
   [junit4]   2> 20040 INFO  (jetty-launcher-15-thread-1) [    ] o.e.j.s.Server Started @22712ms
   [junit4]   2> 20040 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36864}
   [junit4]   2> 20040 ERROR (jetty-launcher-15-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 20040 INFO  (jetty-launcher-15-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 20048 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 20048 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 20048 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 20048 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-08T02:16:44.544Z
   [junit4]   2> 20049 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 20057 INFO  (jetty-launcher-15-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@43b2610e{/solr,null,AVAILABLE}
   [junit4]   2> 20060 INFO  (jetty-launcher-15-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@9d459ce{HTTP/1.1,[http/1.1]}{127.0.0.1:46865}
   [junit4]   2> 20061 INFO  (jetty-launcher-15-thread-2) [    ] o.e.j.s.Server Started @22733ms
   [junit4]   2> 20061 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46865}
   [junit4]   2> 20061 ERROR (jetty-launcher-15-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 20061 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 20062 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 20062 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 20062 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-08T02:16:44.558Z
   [junit4]   2> 20111 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 20129 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 20146 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 20147 INFO  (jetty-launcher-15-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47992/solr
   [junit4]   2> 20184 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 20186 INFO  (jetty-launcher-15-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47992/solr
   [junit4]   2> 20439 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 20442 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36864_solr
   [junit4]   2> 20443 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 20446 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.c.Overseer Overseer (id=97579132799549445-127.0.0.1:36864_solr-n_0000000000) starting
   [junit4]   2> 20448 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46865_solr
   [junit4]   2> 20454 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:46865_solr) [n:127.0.0.1:46865_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 20457 INFO  (zkCallback-26-thread-1-processing-n:127.0.0.1:36864_solr) [n:127.0.0.1:36864_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 20548 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 20569 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 20611 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36864_solr
   [junit4]   2> 20755 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 20757 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
   [junit4]   2> 20760 INFO  (zkCallback-27-thread-1-processing-n:127.0.0.1:46865_solr) [n:127.0.0.1:46865_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 20757 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.s.HadoopAuthPlugin Params: {delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, type=simple, zk-dt-secret-manager.enable=true, signer.secret.provider=zookeeper}
   [junit4]   2> 20790 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.c.f.i.CuratorFrameworkImpl Starting
   [junit4]   2> 20827 INFO  (zkCallback-26-thread-2-processing-n:127.0.0.1:36864_solr) [n:127.0.0.1:36864_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 21682 INFO  (jetty-launcher-15-thread-1-EventThread) [n:127.0.0.1:36864_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 21682 INFO  (jetty-launcher-15-thread-2-EventThread) [n:127.0.0.1:46865_solr    ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
   [junit4]   2> 22055 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 22055 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.h.c.HttpShardHandlerFactory Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@2f94aa11
   [junit4]   2> 22055 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.u.UpdateShardHandler Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@2f94aa11
   [junit4]   2> 22071 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
   [junit4]   2> 22071 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.h.c.HttpShardHandlerFactory Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@65ffb43b
   [junit4]   2> 22071 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.u.UpdateShardHandler Reconfiguring the default client with: org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer@65ffb43b
   [junit4]   2> 22119 INFO  (jetty-launcher-15-thread-2) [n:127.0.0.1:46865_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_4A2C8E19EED5E8D9-001/tempDir-001/node2/.
   [junit4]   2> 22227 INFO  (jetty-launcher-15-thread-1) [n:127.0.0.1:36864_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_4A2C8E19EED5E8D9-001/tempDir-001/node1/.
   [junit4]   2> 22385 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenVerify
   [junit4]   2> 23028 INFO  (qtp1646151721-123) [n:127.0.0.1:46865_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVqrtFasigFaz8DarAEEFLOnvnZITmjjmh6Rm2elIFw9dj5FB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 23029 INFO  (qtp1646151721-123) [n:127.0.0.1:46865_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVqrtFasigFaz8DarAEEFLOnvnZITmjjmh6Rm2elIFw9dj5FB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 23065 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23076 INFO  (qtp243548230-107) [n:127.0.0.1:36864_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVqrtFasigFaz8DarAEEFLOnvnZITmjjmh6Rm2elIFw9dj5FB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 23094 INFO  (qtp243548230-107) [n:127.0.0.1:36864_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVqrtFasigFaz8DarAEEFLOnvnZITmjjmh6Rm2elIFw9dj5FB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=18
   [junit4]   2> 23122 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 23124 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:47992/solr ready
   [junit4]   2> 23131 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23138 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenVerify-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenVerify
   [junit4]   2> 23161 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testZNodePaths
   [junit4]   2> 23198 INFO  (TEST-TestDelegationWithHadoopAuth.testZNodePaths-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testZNodePaths
   [junit4]   2> 23217 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancel
   [junit4]   2> 23282 ERROR (qtp1646151721-122) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_3]
   [junit4]   2> 23317 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23328 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 23330 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:47992/solr ready
   [junit4]   2> 23337 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23367 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 23369 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:47992/solr ready
   [junit4]   2> 23372 ERROR (qtp1646151721-121) [    ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_4]
   [junit4]   2> 23382 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23391 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 23409 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:47992/solr ready
   [junit4]   2> 23416 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23419 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenCancel-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancel
   [junit4]   2> 23442 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenewFail
   [junit4]   2> 23457 WARN  (qtp1646151721-121) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 23457 WARN  (qtp1646151721-121) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 23457 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23460 WARN  (qtp243548230-117) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: bar tries to renew a token with renewer foo
   [junit4]   2> 23461 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23461 WARN  (qtp243548230-117) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 23471 WARN  (qtp1646151721-124) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 23472 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23472 WARN  (qtp1646151721-124) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 23474 WARN  (qtp243548230-118) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
   [junit4]   2> 23475 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23475 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenewFail-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenewFail
   [junit4]   2> 23475 WARN  (qtp243548230-118) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 23499 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenew
   [junit4]   2> 23508 WARN  (qtp1646151721-125) [    ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: foo is trying to renew a token with wrong password
   [junit4]   2> 23509 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23510 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenRenew-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenew
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDelegationWithHadoopAuth -Dtests.method=testDelegationTokenRenew -Dtests.seed=4A2C8E19EED5E8D9 -Dtests.slow=true -Dtests.locale=sl-SI -Dtests.timezone=Asia/Sakhalin -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.04s J1 | TestDelegationWithHadoopAuth.testDelegationTokenRenew <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<200> but was:<403>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4A2C8E19EED5E8D9:7DB77A07D619357D]:0)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.renewDelegationToken(TestDelegationWithHadoopAuth.java:118)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.verifyDelegationTokenRenew(TestDelegationWithHadoopAuth.java:302)
   [junit4]    > 	at org.apache.solr.security.hadoop.TestDelegationWithHadoopAuth.testDelegationTokenRenew(TestDelegationWithHadoopAuth.java:319)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 23512 WARN  (qtp1646151721-125) [    ] o.a.s.s.HadoopAuthPlugin Could not find org.apache.solr.security.authentication.requestcontinues
   [junit4]   2> 23539 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenSolrClient
   [junit4]   2> 23558 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Domain=127.0.0.1; Expires= et, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: et, 01-jan.-1970 00:00:00 GMT
   [junit4]   2> 23568 INFO  (qtp1646151721-121) [n:127.0.0.1:46865_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADZm9vA2ZvbwCKAVqrtFqmigFaz8DepggEFHJ-ODBBNwVRPQy1PT0XUdnPCBhCB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 23569 INFO  (qtp1646151721-121) [n:127.0.0.1:46865_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADZm9vA2ZvbwCKAVqrtFqmigFaz8DepggEFHJ-ODBBNwVRPQy1PT0XUdnPCBhCB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 23570 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationTokenSolrClient-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenSolrClient
   [junit4] IGNOR/A 0.00s J1 | TestDelegationWithHadoopAuth.testDelegationTokenCancelFail
   [junit4]    > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/HADOOP-14044))
   [junit4]   2> 23589 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationOtherOp
   [junit4]   2> 23602 INFO  (qtp1646151721-121) [n:127.0.0.1:46865_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 23602 INFO  (qtp1646151721-121) [n:127.0.0.1:46865_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={op=someSolrOperation&action=LIST&user.name=foo&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 23603 WARN  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[4A2C8E19EED5E8D9]) [    ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=foo&p=foo&t=simple-dt&e=1488975408097&s=clTxnnlSiojeeXs68y1VcaMSkWU="; Domain=127.0.0.1; Expires=Sre, 08-mar.-2017 12:16:48 GMT; HttpOnly". Invalid 'expires' attribute: Sre, 08-mar.-2017 12:16:48 GMT
   [junit4]   2> 23603 INFO  (TEST-TestDelegationWithHadoopAuth.testDelegationOtherOp-seed#[4A2C8E19EED5E8D9]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationOtherOp
   [junit4]   2> 23607 INFO  (jetty-closer-16-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@9d459ce{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 23607 INFO  (jetty-closer-16-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1493911359
   [junit4]   2> 23608 INFO  (jetty-closer-16-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5d9200c4{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 23610 INFO  (jetty-closer-16-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1743706362
   [junit4]   2> 23619 INFO  (jetty-closer-16-thread-2) [    ] o.a.s.c.Overseer Overseer (id=97579132799549445-127.0.0.1:36864_solr-n_0000000000) closing
   [junit4]   2> 23619 INFO  (OverseerStateUpdate-97579132799549445-127.0.0.1:36864_solr-n_0000000000) [n:127.0.0.1:36864_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:36864_solr
   [junit4]   2> 23628 WARN  (zkCallback-27-thread-1-processing-n:127.0.0.1:46865_solr) [n:127.0.0.1:46865_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 23628 INFO  (jetty-closer-16-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 23628 ERROR (Thread[Thread-37,5,TGRP-TestDelegationWithHadoopAuth]) [n:127.0.0.1:46865_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 23647 INFO  (jetty-closer-16-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@43b2610e{/solr,null,UNAVAILABLE}
   [junit4]   2> 25128 WARN  (zkCallback-26-thread-2-processing-n:127.0.0.1:36864_solr) [n:127.0.0.1:36864_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 25128 INFO  (jetty-closer-16-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 25128 ERROR (Thread[Thread-39,5,TGRP-TestDelegationWithHadoopAuth]) [n:127.0.0.1:36864_solr    ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 25134 INFO  (jetty-closer-16-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@359dc0e8{/solr,null,UNAVAILABLE}
   [junit4]   2> 25141 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[4A2C8E19EED5E8D9]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47992 47992
   [junit4]   2> 25192 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
   [junit4]   2> java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 25194 INFO  (Thread-17) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47992 47992
   [junit4]   2> 25196 WARN  (Thread-17) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	2	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_7
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_6
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_8
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_1
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_3
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_2
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_5
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_4
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 	3	/solr/overseer/queue
   [junit4]   2> 	3	/solr/overseer/collection-queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
   [junit4]   2> 
   [junit4]   2> 25197 INFO  (SUITE-TestDelegationWithHadoopAuth-seed#[4A2C8E19EED5E8D9]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestDelegationWithHadoopAuth_4A2C8E19EED5E8D9-001
   [junit4]   2> Mar 08, 2017 2:16:49 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=10, maxDocsPerChunk=579, blockSize=414), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=10, blockSize=414)), sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=sl-SI, timezone=Asia/Sakhalin
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=31913368,total=97320960
   [junit4]   2> NOTE: All tests run in this JVM: [EchoParamsTest, TestEmbeddedSolrServerSchemaAPI, TestObjectReleaseTracker, SuggesterWFSTTest, TestTrieFacet, TestTrie, TestSolrQueryParserResource, TestDelegationWithHadoopAuth]
   [junit4] Completed [10/697 (1!)] on J1 in 6.02s, 8 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 64930 lines...]


[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 710 - Still unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/710/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CdcrBootstrapTest.testBootstrapWithSourceCluster

Error Message:
Document mismatch on target after sync expected:<2000> but was:<1000>

Stack Trace:
java.lang.AssertionError: Document mismatch on target after sync expected:<2000> but was:<1000>
	at __randomizedtesting.SeedInfo.seed([48351F47AE662088:91634E83AD0233C2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.CdcrBootstrapTest.testBootstrapWithSourceCluster(CdcrBootstrapTest.java:232)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12520 lines...]
   [junit4] Suite: org.apache.solr.cloud.CdcrBootstrapTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/init-core-data-001
   [junit4]   2> 1703238 INFO  (SUITE-CdcrBootstrapTest-seed#[48351F47AE662088]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 1703239 INFO  (SUITE-CdcrBootstrapTest-seed#[48351F47AE662088]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1703241 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testBootstrapWithSourceCluster
   [junit4]   2> 1703241 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-target-001
   [junit4]   2> 1703241 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1703241 INFO  (Thread-5683) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1703241 INFO  (Thread-5683) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1703341 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.ZkTestServer start zk server on port:61550
   [junit4]   2> 1703349 INFO  (jetty-launcher-3504-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1703351 INFO  (jetty-launcher-3504-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@69445049{/solr,null,AVAILABLE}
   [junit4]   2> 1703351 INFO  (jetty-launcher-3504-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@729ea8fe{HTTP/1.1,[http/1.1]}{127.0.0.1:44270}
   [junit4]   2> 1703351 INFO  (jetty-launcher-3504-thread-1) [    ] o.e.j.s.Server Started @1708304ms
   [junit4]   2> 1703351 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44270}
   [junit4]   2> 1703352 ERROR (jetty-launcher-3504-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1703352 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 6.5.0
   [junit4]   2> 1703352 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1703352 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1703352 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-07T17:13:01.921Z
   [junit4]   2> 1703354 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1703361 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1703362 INFO  (jetty-launcher-3504-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61550/solr
   [junit4]   2> 1703397 INFO  (jetty-launcher-3504-thread-1) [n:127.0.0.1:44270_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1703398 INFO  (jetty-launcher-3504-thread-1) [n:127.0.0.1:44270_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44270_solr
   [junit4]   2> 1703400 INFO  (jetty-launcher-3504-thread-1) [n:127.0.0.1:44270_solr    ] o.a.s.c.Overseer Overseer (id=97576994852765699-127.0.0.1:44270_solr-n_0000000000) starting
   [junit4]   2> 1703410 INFO  (jetty-launcher-3504-thread-1) [n:127.0.0.1:44270_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44270_solr
   [junit4]   2> 1703413 INFO  (OverseerStateUpdate-97576994852765699-127.0.0.1:44270_solr-n_0000000000) [n:127.0.0.1:44270_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1703476 INFO  (jetty-launcher-3504-thread-1) [n:127.0.0.1:44270_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-target-001/node1/.
   [junit4]   1> Target zkHost = 127.0.0.1:61550/solr
   [junit4]   2> 1703842 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-source-001
   [junit4]   2> 1703842 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1703842 INFO  (Thread-5693) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1703842 INFO  (Thread-5693) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1703845 WARN  (Thread-5693) [    ] o.a.z.j.MBeanRegistry Failed to register MBean StandaloneServer_port-1
   [junit4]   2> 1703845 WARN  (Thread-5693) [    ] o.a.z.s.ZooKeeperServer Failed to register with JMX
   [junit4]   2> javax.management.InstanceAlreadyExistsException: org.apache.ZooKeeperService:name0=StandaloneServer_port-1
   [junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
   [junit4]   2> 	at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96)
   [junit4]   2> 	at org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
   [junit4]   2> 	at org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:308)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
   [junit4]   2> 1703942 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.ZkTestServer start zk server on port:43008
   [junit4]   2> 1703951 INFO  (jetty-launcher-3516-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1703953 INFO  (jetty-launcher-3516-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7b4a0cb9{/solr,null,AVAILABLE}
   [junit4]   2> 1703953 INFO  (jetty-launcher-3516-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2f209abc{HTTP/1.1,[http/1.1]}{127.0.0.1:55396}
   [junit4]   2> 1703953 INFO  (jetty-launcher-3516-thread-1) [    ] o.e.j.s.Server Started @1708907ms
   [junit4]   2> 1703953 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55396}
   [junit4]   2> 1703954 ERROR (jetty-launcher-3516-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1703954 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 6.5.0
   [junit4]   2> 1703954 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1703954 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1703954 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-07T17:13:02.523Z
   [junit4]   2> 1703957 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1703963 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1703964 INFO  (jetty-launcher-3516-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:43008/solr
   [junit4]   2> 1704004 INFO  (jetty-launcher-3516-thread-1) [n:127.0.0.1:55396_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1704006 INFO  (jetty-launcher-3516-thread-1) [n:127.0.0.1:55396_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55396_solr
   [junit4]   2> 1704007 INFO  (jetty-launcher-3516-thread-1) [n:127.0.0.1:55396_solr    ] o.a.s.c.Overseer Overseer (id=97576994892218371-127.0.0.1:55396_solr-n_0000000000) starting
   [junit4]   2> 1704019 INFO  (jetty-launcher-3516-thread-1) [n:127.0.0.1:55396_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55396_solr
   [junit4]   2> 1704022 INFO  (OverseerStateUpdate-97576994892218371-127.0.0.1:55396_solr-n_0000000000) [n:127.0.0.1:55396_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1704130 INFO  (jetty-launcher-3516-thread-1) [n:127.0.0.1:55396_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-source-001/node1/.
   [junit4]   2> 1704546 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1704548 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43008/solr ready
   [junit4]   2> 1704551 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=cdcr-source&name=cdcr-source&action=CREATE&numShards=1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1704555 INFO  (OverseerThreadFactory-7417-thread-1-processing-n:127.0.0.1:55396_solr) [n:127.0.0.1:55396_solr    ] o.a.s.c.CreateCollectionCmd Create collection cdcr-source
   [junit4]   2> 1704663 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=cdcr-source&newCollection=true&name=cdcr-source_shard1_replica1&action=CREATE&numShards=1&collection=cdcr-source&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2
   [junit4]   2> 1704770 INFO  (zkCallback-3523-thread-2-processing-n:127.0.0.1:55396_solr) [n:127.0.0.1:55396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1705675 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1705680 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 1705682 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.IndexSchema [cdcr-source_shard1_replica1] Schema name=minimal
   [junit4]   2> 1705684 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1705685 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/cdcr-source/managed-schema
   [junit4]   2> 1705690 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/cdcr-source/schema.xml to /configs/cdcr-source/schema.xml.bak
   [junit4]   2> 1705691 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-source_shard1_replica1' using configuration from collection cdcr-source
   [junit4]   2> 1705691 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [[cdcr-source_shard1_replica1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-source-001/node1/cdcr-source_shard1_replica1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-source-001/node1/./cdcr-source_shard1_replica1/data/]
   [junit4]   2> 1705761 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 1705762 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1705762 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1705762 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1705764 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@61a62010[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1705765 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-source
   [junit4]   2> 1705765 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-source
   [junit4]   2> 1705766 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-source/managed-schema
   [junit4]   2> 1705767 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1705767 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1705774 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.h.CdcrBufferStateManager Created znode /collections/cdcr-source/cdcr/state/buffer
   [junit4]   2> 1705779 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.h.CdcrProcessStateManager Created znode /collections/cdcr-source/cdcr/state/process
   [junit4]   2> 1705803 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@61a62010[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1705803 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1561231920329654272
   [junit4]   2> 1705812 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1705812 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1705812 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55396/solr/cdcr-source_shard1_replica1/
   [junit4]   2> 1705812 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1705812 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.SyncStrategy http://127.0.0.1:55396/solr/cdcr-source_shard1_replica1/ has no replicas
   [junit4]   2> 1705812 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1705818 INFO  (zkCallback-3523-thread-2-processing-n:127.0.0.1:55396_solr) [n:127.0.0.1:55396_solr    ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-source:shard1
   [junit4]   2> 1705820 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55396/solr/cdcr-source_shard1_replica1/ shard1
   [junit4]   2> 1705931 INFO  (zkCallback-3523-thread-1-processing-n:127.0.0.1:55396_solr) [n:127.0.0.1:55396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1705981 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1705984 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1  x:cdcr-source_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=cdcr-source&newCollection=true&name=cdcr-source_shard1_replica1&action=CREATE&numShards=1&collection=cdcr-source&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2} status=0 QTime=1322
   [junit4]   2> 1705994 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1706090 INFO  (zkCallback-3523-thread-3-processing-n:127.0.0.1:55396_solr) [n:127.0.0.1:55396_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1706558 INFO  (OverseerCollectionConfigSetProcessor-97576994892218371-127.0.0.1:55396_solr-n_0000000000) [n:127.0.0.1:55396_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1706996 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=cdcr-source&name=cdcr-source&action=CREATE&numShards=1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2} status=0 QTime=2446
   [junit4]   1> Adding 10 docs with commit=true, numDocs=100
   [junit4]   2> 1707010 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=8
   [junit4]   2> 1707013 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707014 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707025 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@233b408d[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707025 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707025 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@233b408d[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100)))}
   [junit4]   2> 1707026 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=12
   [junit4]   1> Adding 10 docs with commit=true, numDocs=200
   [junit4]   2> 1707033 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 1707035 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707035 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707046 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2ab39857[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707046 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@2ab39857[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100)))}
   [junit4]   2> 1707046 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707046 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=11
   [junit4]   1> Adding 10 docs with commit=true, numDocs=300
   [junit4]   2> 1707052 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1707055 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707055 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707066 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2b10858d[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707067 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707067 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@2b10858d[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100)))}
   [junit4]   2> 1707067 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=12
   [junit4]   1> Adding 10 docs with commit=true, numDocs=400
   [junit4]   2> 1707072 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1707074 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707074 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707087 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4528277e[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707087 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707087 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@4528277e[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100) Uninverting(_3(6.5.0):C100)))}
   [junit4]   2> 1707087 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=13
   [junit4]   1> Adding 10 docs with commit=true, numDocs=500
   [junit4]   2> 1707093 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1707095 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707095 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707108 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@28e62b72[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707108 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707109 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@28e62b72[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100) Uninverting(_3(6.5.0):C100) Uninverting(_4(6.5.0):C100)))}
   [junit4]   2> 1707109 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=13
   [junit4]   1> Adding 10 docs with commit=true, numDocs=600
   [junit4]   2> 1707114 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1707115 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707115 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707129 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d84da65[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707129 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707129 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@4d84da65[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100) Uninverting(_3(6.5.0):C100) Uninverting(_4(6.5.0):C100) Uninverting(_5(6.5.0):C100)))}
   [junit4]   2> 1707129 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=14
   [junit4]   1> Adding 10 docs with commit=true, numDocs=700
   [junit4]   2> 1707134 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1707136 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707136 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707150 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@315fea2[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707150 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707151 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@315fea2[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100) Uninverting(_3(6.5.0):C100) Uninverting(_4(6.5.0):C100) Uninverting(_5(6.5.0):C100) Uninverting(_6(6.5.0):C100)))}
   [junit4]   2> 1707151 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=15
   [junit4]   1> Adding 10 docs with commit=true, numDocs=800
   [junit4]   2> 1707160 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1707161 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707161 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707176 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@16b02e6[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707176 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707176 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@16b02e6[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100) Uninverting(_3(6.5.0):C100) Uninverting(_4(6.5.0):C100) Uninverting(_5(6.5.0):C100) Uninverting(_6(6.5.0):C100) Uninverting(_7(6.5.0):C100)))}
   [junit4]   2> 1707177 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=15
   [junit4]   1> Adding 10 docs with commit=true, numDocs=900
   [junit4]   2> 1707181 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1707183 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707183 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707198 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@65e7ca2[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707199 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@65e7ca2[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100) Uninverting(_3(6.5.0):C100) Uninverting(_4(6.5.0):C100) Uninverting(_5(6.5.0):C100) Uninverting(_6(6.5.0):C100) Uninverting(_7(6.5.0):C100) Uninverting(_8(6.5.0):C100)))}
   [junit4]   2> 1707199 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707199 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=15
   [junit4]   1> Adding 10 docs with commit=true, numDocs=1000
   [junit4]   2> 1707204 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1707206 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1707206 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1707222 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@27f85fdc[cdcr-source_shard1_replica1] main]
   [junit4]   2> 1707222 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1707223 INFO  (searcherExecutor-7420-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.SolrCore [cdcr-source_shard1_replica1] Registered new searcher Searcher@27f85fdc[cdcr-source_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.5.0):C100) Uninverting(_1(6.5.0):C100) Uninverting(_2(6.5.0):C100) Uninverting(_3(6.5.0):C100) Uninverting(_4(6.5.0):C100) Uninverting(_5(6.5.0):C100) Uninverting(_6(6.5.0):C100) Uninverting(_7(6.5.0):C100) Uninverting(_8(6.5.0):C100) Uninverting(_9(6.5.0):C100)))}
   [junit4]   2> 1707223 INFO  (qtp230876729-19289) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-source:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=17
   [junit4]   2> 1707227 INFO  (qtp230876729-19287) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/select params={q=*:*&_stateVer_=cdcr-source:3&wt=javabin&version=2} hits=1000 status=0 QTime=3
   [junit4]   2> 1707239 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1707240 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:61550/solr ready
   [junit4]   2> 1707242 INFO  (qtp827698803-19244) [n:127.0.0.1:44270_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=cdcr-target&name=cdcr-target&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1707248 INFO  (OverseerThreadFactory-7409-thread-1-processing-n:127.0.0.1:44270_solr) [n:127.0.0.1:44270_solr    ] o.a.s.c.CreateCollectionCmd Create collection cdcr-target
   [junit4]   2> 1707356 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=cdcr-target&newCollection=true&name=cdcr-target_shard1_replica1&action=CREATE&numShards=1&collection=cdcr-target&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1707463 INFO  (zkCallback-3511-thread-1-processing-n:127.0.0.1:44270_solr) [n:127.0.0.1:44270_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1708373 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1708378 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 1708381 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.IndexSchema [cdcr-target_shard1_replica1] Schema name=minimal
   [junit4]   2> 1708383 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1708384 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/cdcr-target/managed-schema
   [junit4]   2> 1708389 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/cdcr-target/schema.xml to /configs/cdcr-target/schema.xml.bak
   [junit4]   2> 1708390 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-target_shard1_replica1' using configuration from collection cdcr-target
   [junit4]   2> 1708390 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [[cdcr-target_shard1_replica1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-target-001/node1/cdcr-target_shard1_replica1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-target-001/node1/./cdcr-target_shard1_replica1/data/]
   [junit4]   2> 1708444 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
   [junit4]   2> 1708445 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1708445 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1708445 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1708446 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@12336a68[cdcr-target_shard1_replica1] main]
   [junit4]   2> 1708449 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-target
   [junit4]   2> 1708449 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-target
   [junit4]   2> 1708449 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-target/managed-schema
   [junit4]   2> 1708450 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1708450 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1708455 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.h.CdcrBufferStateManager Created znode /collections/cdcr-target/cdcr/state/buffer
   [junit4]   2> 1708458 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.h.CdcrProcessStateManager Created znode /collections/cdcr-target/cdcr/state/process
   [junit4]   2> 1708460 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1561231923115720704
   [junit4]   2> 1708460 INFO  (searcherExecutor-7426-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1] Registered new searcher Searcher@12336a68[cdcr-target_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1708469 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1708469 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1708469 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44270/solr/cdcr-target_shard1_replica1/
   [junit4]   2> 1708470 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1708470 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.SyncStrategy http://127.0.0.1:44270/solr/cdcr-target_shard1_replica1/ has no replicas
   [junit4]   2> 1708470 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 1708476 INFO  (zkCallback-3511-thread-3-processing-n:127.0.0.1:44270_solr) [n:127.0.0.1:44270_solr    ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-target:shard1
   [junit4]   2> 1708479 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44270/solr/cdcr-target_shard1_replica1/ shard1
   [junit4]   2> 1708585 INFO  (zkCallback-3511-thread-2-processing-n:127.0.0.1:44270_solr) [n:127.0.0.1:44270_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1708631 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1708634 INFO  (qtp827698803-19248) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1  x:cdcr-target_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=cdcr-target&newCollection=true&name=cdcr-target_shard1_replica1&action=CREATE&numShards=1&collection=cdcr-target&shard=shard1&wt=javabin&version=2} status=0 QTime=1279
   [junit4]   2> 1708641 INFO  (qtp827698803-19244) [n:127.0.0.1:44270_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 1708742 INFO  (zkCallback-3511-thread-1-processing-n:127.0.0.1:44270_solr) [n:127.0.0.1:44270_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1709250 INFO  (OverseerCollectionConfigSetProcessor-97576994852765699-127.0.0.1:44270_solr-n_0000000000) [n:127.0.0.1:44270_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 1709642 INFO  (qtp827698803-19244) [n:127.0.0.1:44270_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=cdcr-target&name=cdcr-target&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=2401
   [junit4]   2> 1709650 INFO  (qtp827698803-19249) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/cdcr params={qt=/cdcr&_stateVer_=cdcr-target:3&action=start&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1709651 INFO  (zkCallback-3511-thread-1-processing-n:127.0.0.1:44270_solr) [n:127.0.0.1:44270_solr    ] o.a.s.h.CdcrProcessStateManager The CDCR process state has changed: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/cdcr/state/process @ cdcr-target:shard1
   [junit4]   2> 1709651 INFO  (zkCallback-3511-thread-1-processing-n:127.0.0.1:44270_solr) [n:127.0.0.1:44270_solr    ] o.a.s.h.CdcrProcessStateManager Received new CDCR process state from watcher: STARTED @ cdcr-target:shard1
   [junit4]   2> 1709656 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1709657 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:61550/solr ready
   [junit4]   2> 1709667 INFO  (qtp827698803-19251) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
   [junit4]   2> 1709668 INFO  (qtp827698803-19251) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 1709674 INFO  (qtp827698803-19250) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/cdcr params={_stateVer_=cdcr-target:3&action=COLLECTIONCHECKPOINT&wt=javabin&version=2} status=0 QTime=13
   [junit4]   2> 1709677 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.h.CdcrReplicatorManager Create new update log reader for target cdcr-target with checkpoint -1 @ cdcr-source:shard1
   [junit4]   2> 1709678 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-target, shard: shard1
   [junit4]   2> 1709686 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.h.CdcrReplicatorManager Submitting bootstrap task to executor
   [junit4]   2> 1709688 INFO  (cdcr-bootstrap-status-3535-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-target shard: shard1 leader: http://127.0.0.1:44270/solr/cdcr-target_shard1_replica1/
   [junit4]   2> 1709701 INFO  (zkCallback-3523-thread-3-processing-n:127.0.0.1:55396_solr) [n:127.0.0.1:55396_solr    ] o.a.s.h.CdcrProcessStateManager The CDCR process state has changed: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/cdcr/state/process @ cdcr-source:shard1
   [junit4]   2> 1709702 INFO  (zkCallback-3523-thread-3-processing-n:127.0.0.1:55396_solr) [n:127.0.0.1:55396_solr    ] o.a.s.h.CdcrProcessStateManager Received new CDCR process state from watcher: STARTED @ cdcr-source:shard1
   [junit4]   2> 1709702 INFO  (qtp827698803-19247) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/cdcr params={qt=/cdcr&masterUrl=http://127.0.0.1:55396/solr/cdcr-source_shard1_replica1/&action=BOOTSTRAP&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 1709705 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1709707 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1709707 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@10c4327f
   [junit4]   2> 1709710 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@63a456f9[cdcr-source_shard1_replica1] realtime]
   [junit4]   2> 1709710 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1709710 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 1709712 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1709712 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Master's generation: 12
   [junit4]   2> 1709712 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Master's version: 1488906788276
   [junit4]   2> 1709712 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1709712 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 1709712 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1709713 INFO  (qtp230876729-19290) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/cdcr params={qt=/cdcr&_stateVer_=cdcr-source:3&action=start&wt=javabin&version=2} status=0 QTime=60
   [junit4]   2> 1709718 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.h.ReplicationHandler Adding tlog files to list: [{size=4649, name=tlog.0000000000000000000.1561231921587945472}, {size=4770, name=tlog.0000000000000000001.1561231921614159872}, {size=4770, name=tlog.0000000000000000002.1561231921636179968}, {size=4770, name=tlog.0000000000000000003.1561231921657151488}, {size=4770, name=tlog.0000000000000000004.1561231921680220160}, {size=4770, name=tlog.0000000000000000005.1561231921701191680}, {size=4770, name=tlog.0000000000000000006.1561231921722163200}, {size=4770, name=tlog.0000000000000000007.1561231921749426176}, {size=4770, name=tlog.0000000000000000008.1561231921772494848}, {size=4770, name=tlog.0000000000000000009.1561231921795563520}]
   [junit4]   2> 1709718 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=4
   [junit4]   2> 1709718 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Number of files in latest index in master: 11
   [junit4]   2> 1709718 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Number of tlog files in master: 10
   [junit4]   2> 1709719 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1709720 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/cdcr params={qt=/cdcr&_stateVer_=cdcr-source:3&action=queues&wt=javabin&version=2} status=0 QTime=0
   [junit4]   1> Cdcr queue response: {responseHeader={status=0,QTime=0},queues={127.0.0.1:61550/solr={cdcr-target={queueSize=1010,lastTimestamp=}}},tlogTotalSize=47579,tlogTotalCount=10,updateLogSynchronizer=stopped}
   [junit4]   2> 1709722 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@6bf09622 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@41ec120f)
   [junit4]   2> 1709723 INFO  (qtp827698803-19247) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor Ignoring commit while not ACTIVE - state: BUFFERING replay: false
   [junit4]   2> 1709723 INFO  (qtp827698803-19247) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/update params={_stateVer_=cdcr-target:3&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1709724 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709729 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709731 INFO  (qtp230876729-19292) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.pos&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709734 INFO  (qtp230876729-19291) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709737 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709739 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709741 INFO  (qtp230876729-19285) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.tim&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709744 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.tip&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709746 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709748 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=_a.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709751 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&file=segments_c&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709751 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Starting download of tlog files from master: [{size=4649, name=tlog.0000000000000000000.1561231921587945472}, {size=4770, name=tlog.0000000000000000001.1561231921614159872}, {size=4770, name=tlog.0000000000000000002.1561231921636179968}, {size=4770, name=tlog.0000000000000000003.1561231921657151488}, {size=4770, name=tlog.0000000000000000004.1561231921680220160}, {size=4770, name=tlog.0000000000000000005.1561231921701191680}, {size=4770, name=tlog.0000000000000000006.1561231921722163200}, {size=4770, name=tlog.0000000000000000007.1561231921749426176}, {size=4770, name=tlog.0000000000000000008.1561231921772494848}, {size=4770, name=tlog.0000000000000000009.1561231921795563520}]
   [junit4]   2> 1709754 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000000.1561231921587945472&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709757 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000001.1561231921614159872&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709759 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000002.1561231921636179968&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709762 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000003.1561231921657151488&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709764 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000004.1561231921680220160&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709766 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000005.1561231921701191680&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709769 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000006.1561231921722163200&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709771 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000007.1561231921749426176&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709775 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000008.1561231921772494848&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709777 INFO  (qtp230876729-19288) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.c.S.Request [cdcr-source_shard1_replica1]  webapp=/solr path=/replication params={generation=12&qt=/replication&checksum=true&tlogFile=tlog.0000000000000000009.1561231921795563520&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1709778 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=92635) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@6bf09622 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@41ec120f)
   [junit4]   2> 1709785 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1709785 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.IndexFetcher Reloading SolrCore cdcr-target_shard1_replica1
   [junit4]   2> 1709786 INFO  (qtp827698803-19247) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b87022c[cdcr-target_shard1_replica1] main]
   [junit4]   2> 1709786 INFO  (qtp827698803-19247) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2} status=0 QTime=62
   [junit4]   2> 1709787 INFO  (cdcr-bootstrap-status-3535-thread-1-processing-n:127.0.0.1:55396_solr x:cdcr-source_shard1_replica1 s:shard1 c:cdcr-source r:core_node1) [n:127.0.0.1:55396_solr c:cdcr-source s:shard1 r:core_node1 x:cdcr-source_shard1_replica1] o.a.s.h.CdcrReplicatorManager CDCR bootstrap running for 1 seconds, sleeping for 2000 ms
   [junit4]   2> 1709788 INFO  (searcherExecutor-7426-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1] Registered new searcher Searcher@4b87022c[cdcr-target_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.5.0):C1000)))}
   [junit4]   2> 1709788 INFO  (qtp827698803-19249) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/select params={q=*:*&_stateVer_=cdcr-target:3&wt=javabin&version=2} hits=1000 status=0 QTime=63
   [junit4]   2> 1709796 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1709801 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.IndexSchema [cdcr-target_shard1_replica1] Schema name=minimal
   [junit4]   2> 1709803 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1709805 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.CoreContainer Reloading SolrCore 'cdcr-target_shard1_replica1' using configuration from collection cdcr-target
   [junit4]   2> 1709805 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [[cdcr-target_shard1_replica1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-target-001/node1/cdcr-target_shard1_replica1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001/cdcr-target-001/node1/./cdcr-target_shard1_replica1/data/]
   [junit4]   2> 1709846 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1709847 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1709848 INFO  (qtp827698803-19246) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.S.Request [cdcr-target_shard1_replica1]  webapp=/solr path=/cdcr params={qt=/cdcr&action=CANCEL_BOOTSTRAP&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1709854 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5c240ce0[cdcr-target_shard1_replica1] main]
   [junit4]   2> 1709856 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-target
   [junit4]   2> 1709857 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-target
   [junit4]   2> 1709857 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-target/managed-schema
   [junit4]   2> 1709857 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 1709858 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1709863 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1709873 INFO  (searcherExecutor-7438-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1] Registered new searcher Searcher@5c240ce0[cdcr-target_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.5.0):C1000)))}
   [junit4]   2> 1709873 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@66f2bd[cdcr-target_shard1_replica1] main]
   [junit4]   2> 1709873 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@613f491a
   [junit4]   2> 1709874 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler Solr core is being closed - shutting down CDCR handler @ cdcr-target:shard1
   [junit4]   2> 1709875 INFO  (searcherExecutor-7438-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1] Registered new searcher Searcher@66f2bd[cdcr-target_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(6.5.0):C1000)))}
   [junit4]   2> 1709875 INFO  (Thread-5704) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.cdcr-target.shard1.replica1
   [junit4]   2> 1709876 INFO  (recoveryExecutor-3509-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler No replay needed.
   [junit4]   2> 1709878 ERROR (updateExecutor-3508-thread-1-processing-n:127.0.0.1:44270_solr x:cdcr-target_shard1_replica1 s:shard1 c:cdcr-target r:core_node1) [n:127.0.0.1:44270_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler Bootstrap operation failed
   [junit4]   2> java.util.concurrent.ExecutionException: java.lang.AssertionError
   [junit4]   2> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
   [junit4]   2> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
   [junit4]   2> 	at org.apache.solr.handler.CdcrRequestHandler.lambda$handleBootstrapAction$0(CdcrRequestHandler.java:643)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.AssertionError
   [junit4]   2> 	at org.apache.solr.handler.Cd

[...truncated too long message...]

erseer/collection-queue-work
   [junit4]   2> 	5	/solr/overseer/queue
   [junit4]   2> 	3	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/live_nodes
   [junit4]   2> 	2	/solr/collections
   [junit4]   2> 
   [junit4]   2> 1857508 INFO  (jetty-closer-3581-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@62818f82{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1857509 INFO  (jetty-closer-3581-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=194718635
   [junit4]   2> 1857512 INFO  (coreCloseExecutor-7561-thread-1) [n:127.0.0.1:50420_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.c.SolrCore [cdcr-target_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@10616ffc
   [junit4]   2> 1857512 INFO  (coreCloseExecutor-7561-thread-1) [n:127.0.0.1:50420_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.h.CdcrRequestHandler Solr core is being closed - shutting down CDCR handler @ cdcr-target:shard1
   [junit4]   2> 1857514 INFO  (coreCloseExecutor-7561-thread-1) [n:127.0.0.1:50420_solr c:cdcr-target s:shard1 r:core_node1 x:cdcr-target_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.cdcr-target.shard1.replica1
   [junit4]   2> 1857514 INFO  (zkCallback-3587-thread-3-processing-n:127.0.0.1:50420_solr) [n:127.0.0.1:50420_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1857516 INFO  (jetty-closer-3581-thread-1) [    ] o.a.s.c.Overseer Overseer (id=97577004109201411-127.0.0.1:50420_solr-n_0000000000) closing
   [junit4]   2> 1857516 INFO  (OverseerStateUpdate-97577004109201411-127.0.0.1:50420_solr-n_0000000000) [n:127.0.0.1:50420_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:50420_solr
   [junit4]   2> 1857518 WARN  (zkCallback-3587-thread-3-processing-n:127.0.0.1:50420_solr) [n:127.0.0.1:50420_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1857520 INFO  (jetty-closer-3581-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1857520 INFO  (jetty-closer-3581-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2138fe70{/solr,null,UNAVAILABLE}
   [junit4]   2> 1857524 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42291 42291
   [junit4]   2> 1857541 INFO  (Thread-5741) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42291 42291
   [junit4]   2> 1857543 WARN  (Thread-5741) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	3	/solr/configs/cdcr-target/managed-schema
   [junit4]   2> 	3	/solr/aliases.json
   [junit4]   2> 	3	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/collections/cdcr-target/leaders/shard1/leader
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/collections/cdcr-target/cdcr/state/process
   [junit4]   2> 	3	/solr/collections/cdcr-target/cdcr/state/buffer
   [junit4]   2> 	3	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	11	/solr/overseer/collection-queue-work
   [junit4]   2> 	5	/solr/overseer/queue
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 	3	/solr/collections
   [junit4]   2> 
   [junit4]   2> 1857543 INFO  (TEST-CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster-seed#[48351F47AE662088]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testBootstrapWithContinousIndexingOnSourceCluster
   [junit4]   2> 1857544 INFO  (SUITE-CdcrBootstrapTest-seed#[48351F47AE662088]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_48351F47AE662088-001
   [junit4]   2> Mar 07, 2017 5:15:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=es-CU, timezone=Europe/Skopje
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=153873864,total=492830720
   [junit4]   2> NOTE: All tests run in this JVM: [BlobRepositoryCloudTest, PolyFieldTest, LeaderInitiatedRecoveryOnCommitTest, UniqFieldsUpdateProcessorFactoryTest, TestCollationFieldDocValues, TestCustomStream, PeerSyncReplicationTest, TestEmbeddedSolrServerConstructors, TestRequestForwarding, TestRTimerTree, TestSolrFieldCacheMBean, TestHdfsCloudBackupRestore, TestClassNameShortening, TestSystemCollAutoCreate, TestDeleteCollectionOnDownNodes, TestTestInjection, TestGroupingSearch, TestFieldCacheSortRandom, SuggesterFSTTest, DateFieldTest, RequiredFieldsTest, XmlUpdateRequestHandlerTest, VMParamsZkACLAndCredentialsProvidersTest, TestRandomDVFaceting, TlogReplayBufferedWhileIndexingTest, TestStressRecovery, EnumFieldTest, TestMinMaxOnMultiValuedField, TestJsonFacetRefinement, DocValuesMultiTest, TestSolr4Spatial2, BitVectorTest, TestBlobHandler, TestBlendedInfixSuggestions, TestDocumentBuilder, HdfsRecoveryZkTest, TestPerFieldSimilarityClassic, DOMUtilTest, TestPostingsSolrHighlighter, JavabinLoaderTest, TestSolrDeletionPolicy1, LeaderFailoverAfterPartitionTest, SaslZkACLProviderTest, TestCloudInspectUtil, TestRawTransformer, TestXIncludeConfig, TestComplexPhraseQParserPlugin, TestFieldCacheWithThreads, TestRandomCollapseQParserPlugin, BlockJoinFacetDistribTest, TestLeaderElectionZkExpiry, TestSQLHandler, TestJsonRequest, TestPhraseSuggestions, HighlighterTest, TestConfigSets, DirectoryFactoryTest, TestSolrCloudWithSecureImpersonation, ResponseLogComponentTest, TestLocalFSCloudBackupRestore, UpdateLogTest, DistribDocExpirationUpdateProcessorTest, HardAutoCommitTest, PreAnalyzedFieldManagedSchemaCloudTest, TestUnifiedSolrHighlighter, ZkStateWriterTest, MetricUtilsTest, TestDefaultSearchFieldResource, DocExpirationUpdateProcessorFactoryTest, DirectSolrConnectionTest, TestPushWriter, DataDrivenBlockJoinTest, DeleteReplicaTest, CdcrReplicationDistributedZkTest, TestMacroExpander, FullHLLTest, TestExactStatsCache, TestMiniSolrCloudClusterSSL, MultiThreadedOCPTest, TestSolrIndexConfig, OverseerModifyCollectionTest, TestManagedResource, TestSchemalessBufferedUpdates, RollingRestartTest, BlockCacheTest, UUIDUpdateProcessorFallbackTest, TestDistributedSearch, TestFieldTypeResource, TestSimpleQParserPlugin, DistributedFacetPivotLongTailTest, TestConfigSetsAPI, TestExtendedDismaxParser, TestConfigSetImmutable, SolrCloudExampleTest, ShardRoutingTest, DisMaxRequestHandlerTest, DeleteLastCustomShardedReplicaTest, TestXmlQParser, QueryElevationComponentTest, TestCollationField, ZkControllerTest, TestNamedUpdateProcessors, TestSolrCoreSnapshots, DistributedFacetExistsSmallTest, SystemInfoHandlerTest, AnalysisErrorHandlingTest, OverseerTest, HttpSolrCallGetCoreTest, DistributedSpellCheckComponentTest, ZkCLITest, StandardRequestHandlerTest, TestIBSimilarityFactory, TestManagedSynonymFilterFactory, TestMissingGroups, TestTolerantUpdateProcessorCloud, TestHighlightDedupGrouping, TestSolrCloudWithDelegationTokens, PathHierarchyTokenizerFactoryTest, QueryEqualityTest, BadComponentTest, SimpleCollectionCreateDeleteTest, JSONWriterTest, TestCollapseQParserPlugin, ConnectionManagerTest, TestHdfsBackupRestoreCore, TestDownShardTolerantSearch, TestLazyCores, TestPerFieldSimilarity, TestWordDelimiterFilterFactory, SolrJmxReporterTest, TestElisionMultitermQuery, DistributedMLTComponentTest, TestStressVersions, TestClusterProperties, DistributedQueryElevationComponentTest, ClassificationUpdateProcessorTest, TestBinaryResponseWriter, TermsComponentTest, DocValuesTest, TestLRUCache, TestExportWriter, TestSolrXml, TestRestManager, TestDistribIDF, AnalysisAfterCoreReloadTest, CustomCollectionTest, MergeStrategyTest, StatelessScriptUpdateProcessorFactoryTest, ZkNodePropsTest, SolrRequestParserTest, SolrIndexMetricsTest, TestAuthenticationFramework, TestInitQParser, TestCharFilters, HdfsSyncSliceTest, TestBulkSchemaConcurrent, MigrateRouteKeyTest, TestReRankQParserPlugin, TestConfigsApi, DistributedQueryComponentCustomSortTest, TestFastLRUCache, JsonValidatorTest, CoreMergeIndexesAdminHandlerTest, TestHashPartitioner, HdfsLockFactoryTest, CoreAdminCreateDiscoverTest, DeleteStatusTest, HdfsWriteToMultipleCollectionsTest, JSONWriterTest, OpenCloseCoreStressTest, RecoveryZkTest, TestReplicationHandler, AutoCommitTest, TestFaceting, TestRecovery, TestStressReorder, TestCoreContainer, SimpleFacetsTest, StatsComponentTest, PeerSyncTest, DirectUpdateHandlerTest, SoftAutoCommitTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, SOLR749Test, TestQuerySenderListener, UpdateParamsTest, AlternateDirectoryTest, CopyFieldTest, SolrIndexConfigTest, MultiTermTest, SampleTest, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, OutputWriterTest, TestSolrCoreProperties, TestLuceneMatchVersion, SpellPossibilityIteratorTest, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestSweetSpotSimilarityFactory, TestBM25SimilarityFactory, TimeZoneUtilsTest, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, BaseCdcrDistributedZkTest, CdcrBootstrapTest]
   [junit4] Completed [497/697 (1!)] on J0 in 155.34s, 3 tests, 1 failure <<< FAILURES!

[...truncated 63340 lines...]


[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 709 - Failure!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/709/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 68150 lines...]
-ecj-javadoc-lint-src:
    [mkdir] Created dir: /var/tmp/ecj775381538
 [ecj-lint] Compiling 1062 source files to /var/tmp/ecj775381538
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: /export/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] ----------
 [ecj-lint] 1. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/cloud/Assign.java (at line 101)
 [ecj-lint] 	Collections.sort(shardIdNames, (o1, o2) -> {
 [ecj-lint] 	                               ^^^^^^^^^^^
 [ecj-lint] (Recovered) Internal inconsistency detected during lambda shape analysis
 [ecj-lint] ----------
 [ecj-lint] 2. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/cloud/Assign.java (at line 101)
 [ecj-lint] 	Collections.sort(shardIdNames, (o1, o2) -> {
 [ecj-lint] 	                               ^^^^^^^^^^^
 [ecj-lint] (Recovered) Internal inconsistency detected during lambda shape analysis
 [ecj-lint] ----------
 [ecj-lint] 3. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/cloud/Assign.java (at line 101)
 [ecj-lint] 	Collections.sort(shardIdNames, (o1, o2) -> {
 [ecj-lint] 	                               ^^^^^^^^^^^
 [ecj-lint] (Recovered) Internal inconsistency detected during lambda shape analysis
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 4. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/cloud/rule/ReplicaAssigner.java (at line 212)
 [ecj-lint] 	Collections.sort(sortedLiveNodes, (n1, n2) -> {
 [ecj-lint] 	                                  ^^^^^^^^^^^
 [ecj-lint] (Recovered) Internal inconsistency detected during lambda shape analysis
 [ecj-lint] ----------
 [ecj-lint] 5. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/cloud/rule/ReplicaAssigner.java (at line 212)
 [ecj-lint] 	Collections.sort(sortedLiveNodes, (n1, n2) -> {
 [ecj-lint] 	                                  ^^^^^^^^^^^
 [ecj-lint] (Recovered) Internal inconsistency detected during lambda shape analysis
 [ecj-lint] ----------
 [ecj-lint] 6. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/cloud/rule/ReplicaAssigner.java (at line 212)
 [ecj-lint] 	Collections.sort(sortedLiveNodes, (n1, n2) -> {
 [ecj-lint] 	                                  ^^^^^^^^^^^
 [ecj-lint] (Recovered) Internal inconsistency detected during lambda shape analysis
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 7. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/core/CoreContainer.java (at line 917)
 [ecj-lint] 	core = new SolrCore(dcore, coreConfig);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'core' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 8. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/core/HdfsDirectoryFactory.java (at line 233)
 [ecj-lint] 	dir = new BlockDirectory(path, hdfsDir, cache, null, blockCacheReadEnabled, false, cacheMerges, cacheReadOnce);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'dir' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 9. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 120)
 [ecj-lint] 	reader = cfiltfac.create(reader);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'reader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] 10. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java (at line 144)
 [ecj-lint] 	return namedList;
 [ecj-lint] 	^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'listBasedTokenStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 11. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/handler/ClassifyStream.java (at line 88)
 [ecj-lint] 	SolrCore solrCore = (SolrCore) solrCoreObj;
 [ecj-lint] 	         ^^^^^^^^
 [ecj-lint] Resource leak: 'solrCore' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 12. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java (at line 1236)
 [ecj-lint] 	DirectoryReader reader = s==null ? null : s.get().getIndexReader();
 [ecj-lint] 	                ^^^^^^
 [ecj-lint] Resource leak: 'reader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 13. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/handler/component/ExpandComponent.java (at line 222)
 [ecj-lint] 	UninvertingReader uninvertingReader = new UninvertingReader(new ReaderWrapper(searcher.getSlowAtomicReader(), field), mapping);
 [ecj-lint] 	                  ^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'uninvertingReader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 14. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 507)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zk, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] 15. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/handler/sql/SolrTable.java (at line 719)
 [ecj-lint] 	ParallelStream parallelStream = new ParallelStream(zkHost, collection, tupleStream, numWorkers, comp);
 [ecj-lint] 	               ^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'parallelStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 16. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java (at line 554)
 [ecj-lint] 	tvWindowStream = new OffsetWindowTokenFilter(tvStream);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'tvWindowStream' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 17. ERROR in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/request/NumericFacets.java (at line 55)
 [ecj-lint] 	import org.apache.solr.util.DateMathParser;
 [ecj-lint] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] The import org.apache.solr.util.DateMathParser is never used
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 18. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/request/SimpleFacets.java (at line 920)
 [ecj-lint] 	fastForRandomSet = new HashDocSet(sset.getDocs(), 0, sset.size());
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'fastForRandomSet' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 19. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/response/BinaryResponseWriter.java (at line 51)
 [ecj-lint] 	new JavaBinCodec(resolver).setWritableDocFields(resolver).marshal(response.getValues(), out);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 20. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/response/BinaryResponseWriter.java (at line 163)
 [ecj-lint] 	new JavaBinCodec(resolver).setWritableDocFields(resolver).marshal(rsp.getValues(), out);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 21. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/response/BinaryResponseWriter.java (at line 166)
 [ecj-lint] 	return (NamedList<Object>) new JavaBinCodec(resolver).unmarshal(in);
 [ecj-lint] 	                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 22. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java (at line 33)
 [ecj-lint] 	new SmileWriter(out, request, response).writeResponse();
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 23. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/schema/OpenExchangeRatesOrgProvider.java (at line 146)
 [ecj-lint] 	ratesJsonStream = resourceLoader.openResource(ratesFileLocation);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'ratesJsonStream' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 24. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java (at line 1185)
 [ecj-lint] 	UninvertingReader uninvertingReader = new UninvertingReader(new ReaderWrapper(searcher.getSlowAtomicReader(), collapseField), mapping);
 [ecj-lint] 	                  ^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'uninvertingReader' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 25. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/CursorMark.java (at line 188)
 [ecj-lint] 	pieces = (List<Object>) new JavaBinCodec().unmarshal(in);
 [ecj-lint] 	                        ^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] 26. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/CursorMark.java (at line 265)
 [ecj-lint] 	new JavaBinCodec().marshal(marshalledValues, out);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: '<unassigned Closeable value>' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 27. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 145)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 28. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 154)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] 29. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/DocSlice.java (at line 163)
 [ecj-lint] 	HashDocSet h = new HashDocSet(docs,offset,len);
 [ecj-lint] 	           ^
 [ecj-lint] Resource leak: 'h' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 30. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/function/OrdFieldSource.java (at line 78)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 31. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/search/function/ReverseOrdFieldSource.java (at line 78)
 [ecj-lint] 	SolrIndexSearcher is = (SolrIndexSearcher) o;
 [ecj-lint] 	                  ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 32. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java (at line 263)
 [ecj-lint] 	throw new SolrException(ErrorCode.SERVER_ERROR, "Error occurred while loading solr.xml from zookeeper", e);
 [ecj-lint] 	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [ecj-lint] Resource leak: 'loader' is not closed at this location
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 33. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/CdcrTransactionLog.java (at line 98)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 34. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 188)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 35. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/HdfsTransactionLog.java (at line 258)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 36. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 226)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 37. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 226)
 [ecj-lint] 	ChannelFastInputStream is = new ChannelFastInputStream(channel, pos);
 [ecj-lint] 	                       ^^
 [ecj-lint] Resource leak: 'is' is never closed
 [ecj-lint] ----------
 [ecj-lint] 38. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 258)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 39. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 273)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] 40. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/TransactionLog.java (at line 527)
 [ecj-lint] 	LogCodec codec = new LogCodec(resolver);
 [ecj-lint] 	         ^^^^^
 [ecj-lint] Resource leak: 'codec' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 41. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/update/processor/UpdateRequestProcessor.java (at line 85)
 [ecj-lint] 	UpdateRequestProcessor p = this;
 [ecj-lint] 	                       ^
 [ecj-lint] Resource leak: 'p' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 42. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] 43. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/util/SimplePostTool.java (at line 655)
 [ecj-lint] 	BAOS bos =  new BAOS();
 [ecj-lint] 	     ^^^
 [ecj-lint] Resource leak: 'bos' is never closed
 [ecj-lint] ----------
 [ecj-lint] ----------
 [ecj-lint] 44. WARNING in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/java/org/apache/solr/util/SolrLogLayout.java (at line 149)
 [ecj-lint] 	SolrQueryRequest req = requestInfo == null ? null : requestInfo.getReq();
 [ecj-lint] 	                 ^^^
 [ecj-lint] Resource leak: 'req' is never closed
 [ecj-lint] ----------
 [ecj-lint] 44 problems (1 error, 43 warnings)

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:775: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:101: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build.xml:686: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/lucene/common-build.xml:2004: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/lucene/common-build.xml:2043: Compile failed; see the compiler error output for details.

Total time: 92 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 708 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/708/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@3c6d11ea (not leader) wrong [docid] for SolrDocument{id=25, id_field_copy_that_does_not_support_in_place_update_s=25, title_s=title25, id_i=25, inplace_updatable_float=101.0, _version_=1561158095162310656, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=274} expected:<247> but was:<274>

Stack Trace:
java.lang.AssertionError: 'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@3c6d11ea (not leader) wrong [docid] for SolrDocument{id=25, id_field_copy_that_does_not_support_in_place_update_s=25, title_s=title25, id_i=25, inplace_updatable_float=101.0, _version_=1561158095162310656, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=274} expected:<247> but was:<274>
	at __randomizedtesting.SeedInfo.seed([B0D418BB606CC73A:38802761CE90AAC2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesInResults(TestInPlaceUpdatesDistrib.java:442)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesAgainstAllClients(TestInPlaceUpdatesDistrib.java:413)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:321)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11309 lines...]
   [junit4] Suite: org.apache.solr.update.TestInPlaceUpdatesDistrib
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/init-core-data-001
   [junit4]   2> 580565 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 580568 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 580568 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 580571 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 580573 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 580611 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 580612 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 580628 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=inplace-updates
   [junit4]   2> 580631 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 580637 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 580671 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 580673 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 580673 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 580677 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 580677 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 580677 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 580678 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/lib, /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 580712 WARN  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 580713 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 580724 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 580728 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 580730 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 580730 WARN  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 580730 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 580730 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/init-core-data-001/]
   [junit4]   2> 580731 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 580732 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 580740 WARN  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 580820 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 580820 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 580823 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 580823 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 580824 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=83.904296875, floorSegmentMB=0.44921875, forceMergeDeletesPctAllowed=1.3851973279627872, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3359084929199888
   [junit4]   2> 580825 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4ee77ba[collection1] main]
   [junit4]   2> 580825 WARN  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 580825 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 580826 INFO  (searcherExecutor-1670-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4ee77ba[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 580826 INFO  (coreLoadExecutor-1669-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1561158068190838784
   [junit4]   2> 580830 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 580834 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 580834 INFO  (Thread-2453) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 580834 INFO  (Thread-2453) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 580934 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ZkTestServer start zk server on port:61894
   [junit4]   2> 580954 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 580957 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml to /configs/conf1/schema.xml
   [junit4]   2> 580959 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 580962 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 580964 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 580966 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 580968 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 580970 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 580973 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 580975 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 580977 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 581119 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/control-001/cores/collection1
   [junit4]   2> 581120 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 581122 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5fb715c5{/,null,AVAILABLE}
   [junit4]   2> 581122 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4f198473{HTTP/1.1,[http/1.1]}{127.0.0.1:58729}
   [junit4]   2> 581122 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server Started @584668ms
   [junit4]   2> 581122 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/tempDir-001/control/data, hostContext=/, hostPort=58729, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/control-001/cores}
   [junit4]   2> 581123 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 581123 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 581123 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 581123 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 581124 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-06T21:39:13.782Z
   [junit4]   2> 581128 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 581128 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/control-001/solr.xml
   [junit4]   2> 581136 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 581136 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61894/solr
   [junit4]   2> 581177 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:58729_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 581179 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:58729_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58729_
   [junit4]   2> 581181 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:58729_    ] o.a.s.c.Overseer Overseer (id=97572379262582788-127.0.0.1:58729_-n_0000000000) starting
   [junit4]   2> 581193 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:58729_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58729_
   [junit4]   2> 581197 INFO  (zkCallback-625-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 581264 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:58729_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/control-001/cores
   [junit4]   2> 581264 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:58729_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 581269 INFO  (OverseerStateUpdate-97572379262582788-127.0.0.1:58729_-n_0000000000) [n:127.0.0.1:58729_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 582285 WARN  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 582286 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 582299 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 582302 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 582304 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 582304 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/control-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/control-001/cores/collection1/data/]
   [junit4]   2> 582305 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 582305 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 582311 WARN  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 582360 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 582360 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 582364 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 582364 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 582365 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=83.904296875, floorSegmentMB=0.44921875, forceMergeDeletesPctAllowed=1.3851973279627872, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3359084929199888
   [junit4]   2> 582366 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6011da5c[collection1] main]
   [junit4]   2> 582368 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 582368 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 582369 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 582370 INFO  (searcherExecutor-1683-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6011da5c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 582370 INFO  (coreLoadExecutor-1682-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1561158069809840128
   [junit4]   2> 582380 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 582380 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 582380 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:58729/collection1/
   [junit4]   2> 582380 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 582380 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:58729/collection1/ has no replicas
   [junit4]   2> 582380 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 582386 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:58729/collection1/ shard1
   [junit4]   2> 582437 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 582439 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:61894/solr ready
   [junit4]   2> 582439 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 582440 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 582487 INFO  (coreZkRegister-1675-thread-1-processing-n:127.0.0.1:58729_ x:collection1 c:control_collection) [n:127.0.0.1:58729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 582637 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-1-001/cores/collection1
   [junit4]   2> 582637 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-1-001
   [junit4]   2> 582638 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 582640 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@e26a0e0{/,null,AVAILABLE}
   [junit4]   2> 582640 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7b782eac{HTTP/1.1,[http/1.1]}{127.0.0.1:37969}
   [junit4]   2> 582640 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server Started @586187ms
   [junit4]   2> 582640 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=37969, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-1-001/cores}
   [junit4]   2> 582641 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 582642 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 582642 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 582642 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 582642 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-06T21:39:15.300Z
   [junit4]   2> 582648 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 582648 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-1-001/solr.xml
   [junit4]   2> 582654 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 582656 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61894/solr
   [junit4]   2> 582674 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 582679 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:37969_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 582684 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:37969_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37969_
   [junit4]   2> 582688 INFO  (zkCallback-625-thread-2-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 582690 INFO  (zkCallback-629-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 582692 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 582832 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:37969_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-1-001/cores
   [junit4]   2> 582833 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:37969_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 582836 INFO  (OverseerStateUpdate-97572379262582788-127.0.0.1:58729_-n_0000000000) [n:127.0.0.1:58729_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 582942 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 583857 WARN  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 583858 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 583873 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 583878 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 583881 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 583882 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-1-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 583882 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 583884 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 583891 WARN  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 583989 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 583989 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 583992 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 583992 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 583994 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=83.904296875, floorSegmentMB=0.44921875, forceMergeDeletesPctAllowed=1.3851973279627872, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3359084929199888
   [junit4]   2> 583994 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1fb1e8a4[collection1] main]
   [junit4]   2> 583996 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 583997 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 583997 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 583999 INFO  (searcherExecutor-1694-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1fb1e8a4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 583999 INFO  (coreLoadExecutor-1693-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1561158071517970432
   [junit4]   2> 584019 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 584020 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 584020 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37969/collection1/
   [junit4]   2> 584020 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 584020 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:37969/collection1/ has no replicas
   [junit4]   2> 584021 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 584032 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37969/collection1/ shard1
   [junit4]   2> 584138 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 584185 INFO  (coreZkRegister-1688-thread-1-processing-n:127.0.0.1:37969_ x:collection1 c:collection1) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 584258 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-2-001/cores/collection1
   [junit4]   2> 584260 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-2-001
   [junit4]   2> 584261 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 584263 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3e039558{/,null,AVAILABLE}
   [junit4]   2> 584265 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6af99c69{HTTP/1.1,[http/1.1]}{127.0.0.1:48265}
   [junit4]   2> 584265 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server Started @587813ms
   [junit4]   2> 584265 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=48265, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-2-001/cores}
   [junit4]   2> 584265 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 584267 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 584267 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 584267 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 584267 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-06T21:39:16.925Z
   [junit4]   2> 584274 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 584274 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-2-001/solr.xml
   [junit4]   2> 584282 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 584284 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61894/solr
   [junit4]   2> 584294 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 584309 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 584314 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:48265_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 584319 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:48265_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48265_
   [junit4]   2> 584323 INFO  (zkCallback-625-thread-1-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584325 INFO  (zkCallback-629-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584325 INFO  (zkCallback-635-thread-1-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584328 INFO  (zkCallback-641-thread-2-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 584429 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 584447 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:48265_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-2-001/cores
   [junit4]   2> 584447 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:48265_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 584451 INFO  (OverseerStateUpdate-97572379262582788-127.0.0.1:58729_-n_0000000000) [n:127.0.0.1:58729_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 584562 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 584563 INFO  (zkCallback-641-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 585499 WARN  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 585500 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 585513 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 585517 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 585518 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 585519 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-2-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 585519 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 585520 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 585525 WARN  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 585588 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 585589 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 585591 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 585591 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 585592 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=83.904296875, floorSegmentMB=0.44921875, forceMergeDeletesPctAllowed=1.3851973279627872, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3359084929199888
   [junit4]   2> 585592 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1420f610[collection1] main]
   [junit4]   2> 585594 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 585595 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 585595 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 585597 INFO  (searcherExecutor-1705-thread-1-processing-n:127.0.0.1:48265_ x:collection1 c:collection1) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1420f610[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 585598 INFO  (coreLoadExecutor-1704-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1561158073194643456
   [junit4]   2> 585605 INFO  (coreZkRegister-1699-thread-1-processing-n:127.0.0.1:48265_ x:collection1 c:collection1) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 585606 INFO  (updateExecutor-638-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 585607 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 585607 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 585607 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 585607 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 585607 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:37969/collection1/] and I am [http://127.0.0.1:48265/collection1/]
   [junit4]   2> 585613 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:37969]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:48265_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 585616 INFO  (qtp768259566-5502) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 585632 INFO  (qtp768259566-5502) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 585632 INFO  (qtp768259566-5502) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:48265_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:48265","node_name":"127.0.0.1:48265_","state":"down"}
   [junit4]   2> 585733 INFO  (zkCallback-641-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 585733 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 585838 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-3-001/cores/collection1
   [junit4]   2> 585839 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-3-001
   [junit4]   2> 585840 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 585843 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36675eb1{/,null,AVAILABLE}
   [junit4]   2> 585843 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@14c81916{HTTP/1.1,[http/1.1]}{127.0.0.1:40579}
   [junit4]   2> 585843 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.Server Started @589392ms
   [junit4]   2> 585843 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=40579, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-3-001/cores}
   [junit4]   2> 585843 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 585844 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 585844 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 585844 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 585845 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-06T21:39:18.502Z
   [junit4]   2> 585851 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 585851 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-3-001/solr.xml
   [junit4]   2> 585858 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 585859 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61894/solr
   [junit4]   2> 585880 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:40579_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 585886 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:40579_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 585890 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:40579_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40579_
   [junit4]   2> 585893 INFO  (zkCallback-629-thread-2) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 585895 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 585895 INFO  (zkCallback-641-thread-1-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 585896 INFO  (zkCallback-625-thread-3-processing-n:127.0.0.1:58729_) [n:127.0.0.1:58729_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 585900 INFO  (zkCallback-648-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 586003 INFO  (zkCallback-641-thread-2-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 586003 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 586122 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:40579_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-3-001/cores
   [junit4]   2> 586122 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [n:127.0.0.1:40579_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 586127 INFO  (OverseerStateUpdate-97572379262582788-127.0.0.1:58729_-n_0000000000) [n:127.0.0.1:58729_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 586231 INFO  (zkCallback-641-thread-2-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 586231 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 586231 INFO  (zkCallback-648-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 586633 INFO  (qtp768259566-5502) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:48265_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:48265","node_name":"127.0.0.1:48265_","state":"recovering"}
   [junit4]   2> 586633 INFO  (qtp768259566-5502) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 586633 INFO  (qtp768259566-5502) [n:127.0.0.1:37969_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:48265_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1017
   [junit4]   2> 587134 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:37969/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 587135 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:48265 START replicas=[http://127.0.0.1:37969/collection1/] nUpdates=100
   [junit4]   2> 587139 INFO  (qtp768259566-5507) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 587139 INFO  (qtp768259566-5507) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:2.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 587143 INFO  (recoveryExecutor-639-thread-1-processing-n:127.0.0.1:48265_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 587153 WARN  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 587154 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 587193 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 587196 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 587198 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 587198 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-3-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 587198 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1450c14b
   [junit4]   2> 587199 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 587205 WARN  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 587252 INFO  (zkCallback-648-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 587253 INFO  (zkCallback-641-thread-2-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 587254 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 587265 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 587265 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 587269 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 587269 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 587269 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=83.904296875, floorSegmentMB=0.44921875, forceMergeDeletesPctAllowed=1.3851973279627872, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3359084929199888
   [junit4]   2> 587270 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@12113d4c[collection1] main]
   [junit4]   2> 587271 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 587272 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 587272 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 587273 INFO  (coreLoadExecutor-1715-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1561158074951008256
   [junit4]   2> 587276 INFO  (searcherExecutor-1716-thread-1-processing-n:127.0.0.1:40579_ x:collection1 c:collection1) [n:127.0.0.1:40579_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@12113d4c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 587283 INFO  (coreZkRegister-1710-thread-1-processing-n:127.0.0.1:40579_ x:collection1 c:collection1) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 587283 INFO  (updateExecutor-645-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 587284 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 587285 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 587285 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 587285 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 587286 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:37969/collection1/] and I am [http://127.0.0.1:40579/collection1/]
   [junit4]   2> 587292 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:37969]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:40579_&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 587296 INFO  (qtp768259566-5509) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 587298 INFO  (qtp768259566-5509) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 587298 INFO  (qtp768259566-5509) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:40579_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:40579","node_name":"127.0.0.1:40579_","state":"down"}
   [junit4]   2> 587352 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 587353 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
   [junit4]   1> -
   [junit4]   1> replica:core_node1 rstate:active live:true
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node3 rstate:down live:true
   [junit4]   2> 587395 INFO  (zkCallback-635-thread-2-processing-n:127.0.0.1:37969_) [n:127.0.0.1:37969_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 587395 INFO  (zkCallback-648-thread-1-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 587396 INFO  (zkCallback-641-thread-2-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 588305 INFO  (qtp768259566-5509) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:40579_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:40579","node_name":"127.0.0.1:40579_","state":"recovering"}
   [junit4]   2> 588305 INFO  (qtp768259566-5509) [n:127.0.0.1:37969_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 588305 INFO  (qtp768259566-5509) [n:127.0.0.1:37969_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:40579_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1009
   [junit4]   1> -
   [junit4]   1> replica:core_node1 rstate:active live:true
   [junit4]   1> replica:core_node2 rstate:active live:true
   [junit4]   1> replica:core_node3 rstate:recovering live:true
   [junit4]   2> 588809 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:37969/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 588810 INFO  (recoveryExecutor-646-thread-1-processing-n:127.0.0.1:40579_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:40579 START replicas=[http://127.0.0.1:37969/collection1/] nUpdates=100
   [junit4]   2> 588813 INFO  (qtp768259566-5503) [n:127.0.0.1:37969_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersi

[...truncated too long message...]

8265_) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@767dab0a
   [junit4]   2> 616135 INFO  (zkCallback-648-thread-3-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40579/collection1/
   [junit4]   2> 616135 INFO  (zkCallback-648-thread-3-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 616135 INFO  (zkCallback-648-thread-3-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40579/collection1/ has no replicas
   [junit4]   2> 616139 INFO  (zkCallback-648-thread-3-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40579/collection1/ shard1
   [junit4]   2> 616141 INFO  (zkCallback-648-thread-2-processing-n:127.0.0.1:40579_) [n:127.0.0.1:40579_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 616163 INFO  (zkCallback-641-thread-3-processing-n:127.0.0.1:48265_) [n:127.0.0.1:48265_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 616163 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 616165 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@6af99c69{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 616166 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3e039558{/,null,UNAVAILABLE}
   [junit4]   2> 616167 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 40579
   [junit4]   2> 616168 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=550357108
   [junit4]   2> 616173 INFO  (coreCloseExecutor-1736-thread-1) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4aad11d8
   [junit4]   2> 616206 INFO  (coreCloseExecutor-1736-thread-1) [n:127.0.0.1:40579_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 616208 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.Overseer Overseer (id=97572379262582799-127.0.0.1:40579_-n_0000000003) closing
   [junit4]   2> 616209 INFO  (OverseerStateUpdate-97572379262582799-127.0.0.1:40579_-n_0000000003) [n:127.0.0.1:40579_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:40579_
   [junit4]   2> 616214 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 616218 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@14c81916{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 616218 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@36675eb1{/,null,UNAVAILABLE}
   [junit4]   2> 616222 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[B0D418BB606CC73A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:61894 61894
   [junit4]   2> 616235 INFO  (Thread-2453) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:61894 61894
   [junit4]   2> 616237 WARN  (Thread-2453) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/aliases.json
   [junit4]   2> 	5	/solr/clusterprops.json
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 	4	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	36	/solr/overseer/collection-queue-work
   [junit4]   2> 	31	/solr/overseer/queue
   [junit4]   2> 	9	/solr/overseer/queue-work
   [junit4]   2> 	5	/solr/live_nodes
   [junit4]   2> 	5	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestInPlaceUpdatesDistrib -Dtests.method=test -Dtests.seed=B0D418BB606CC73A -Dtests.slow=true -Dtests.locale=zh-SG -Dtests.timezone=Asia/Tel_Aviv -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 35.4s J0 | TestInPlaceUpdatesDistrib.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: 'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@3c6d11ea (not leader) wrong [docid] for SolrDocument{id=25, id_field_copy_that_does_not_support_in_place_update_s=25, title_s=title25, id_i=25, inplace_updatable_float=101.0, _version_=1561158095162310656, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=274} expected:<247> but was:<274>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B0D418BB606CC73A:38802761CE90AAC2]:0)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesInResults(TestInPlaceUpdatesDistrib.java:442)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesAgainstAllClients(TestInPlaceUpdatesDistrib.java:413)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:321)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:140)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 616241 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 616241 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1991905084
   [junit4]   2> 616241 INFO  (coreCloseExecutor-1738-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@12dc84bb
   [junit4]   2> 616256 INFO  (coreCloseExecutor-1738-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 616260 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[B0D418BB606CC73A]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/test/J0/temp/solr.update.TestInPlaceUpdatesDistrib_B0D418BB606CC73A-001
   [junit4]   2> Mar 06, 2017 9:39:48 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=8423, maxDocsPerChunk=223, blockSize=4), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=8423, blockSize=4)), sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=zh-SG, timezone=Asia/Tel_Aviv
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=81291608,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [StandardRequestHandlerTest, DocValuesMultiTest, SolrMetricReporterTest, TestCollapseQParserPlugin, HdfsRecoveryZkTest, TestCharFilters, TestExtendedDismaxParser, TestSolrQueryParserDefaultOperatorResource, TestFastOutputStream, TestSolrIndexConfig, SoftAutoCommitTest, SuggesterFSTTest, TestStressVersions, BlockCacheTest, QueryEqualityTest, TestCSVLoader, TestNumericTerms32, RecoveryZkTest, DateRangeFieldTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestBlobHandler, SolrGangliaReporterTest, TestTestInjection, TestHdfsCloudBackupRestore, TestBM25SimilarityFactory, TestReqParamsAPI, LeaderFailoverAfterPartitionTest, TestXmlQParser, TestHdfsBackupRestoreCore, DirectSolrConnectionTest, TestOverriddenPrefixQueryForCustomFieldType, TestInitQParser, SimpleCollectionCreateDeleteTest, TestSchemaResource, HdfsLockFactoryTest, DOMUtilTest, TestGraphMLResponseWriter, TestBulkSchemaConcurrent, PeerSyncTest, DirectoryFactoryTest, DisMaxRequestHandlerTest, JSONWriterTest, SchemaVersionSpecificBehaviorTest, TestClusterStateMutator, TestJsonFacetRefinement, TestCollationFieldDocValues, UnloadDistributedZkTest, FullHLLTest, DistributedDebugComponentTest, ReplicationFactorTest, TestSolrConfigHandler, TestClusterProperties, OverseerTest, TestRebalanceLeaders, TestInPlaceUpdatesDistrib]
   [junit4] Completed [143/697 (1!)] on J0 in 35.98s, 1 test, 1 failure <<< FAILURES!

[...truncated 64541 lines...]