You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2017/02/23 05:58:47 UTC

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1245 - Unstable

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1245/

2 tests failed.
FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
	at __randomizedtesting.SeedInfo.seed([BFA67DA57E6DA6E2:4809B48E83E44CC9]:0)
	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:310)
	at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:286)
	at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:226)
	at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:108)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:693)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:70)
	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:308)
	... 42 more


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1487826451314,generation=2,filelist=[_n27.fdt, _n27.fdx, _n27.fnm, _n27.nvd, _n27.nvm, _n27.si, _n27_Lucene50_0.doc, _n27_Lucene50_0.tim, _n27_Lucene50_0.tip, _n3p.cfe, _n3p.cfs, _n3p.si, _n3q.cfe, _n3q.cfs, _n3q.si, _n3r.cfe, _n3r.cfs, _n3r.si, _n3s.cfe, _n3s.cfs, _n3s.si, _n3t.cfe, _n3t.cfs, _n3t.si, _n57.cfe, _n57.cfs, _n57.si, _n58.cfe, _n58.cfs, _n58.si, _n59.cfe, _n59.cfs, _n59.si, _n5a.cfe, _n5a.cfs, _n5a.si, segments_2]}]> but was:<[{indexVersion=1487826451314,generation=2,filelist=[_n27.fdt, _n27.fdx, _n27.fnm, _n27.nvd, _n27.nvm, _n27.si, _n27_Lucene50_0.doc, _n27_Lucene50_0.tim, _n27_Lucene50_0.tip, _n3p.cfe, _n3p.cfs, _n3p.si, _n3q.cfe, _n3q.cfs, _n3q.si, _n3r.cfe, _n3r.cfs, _n3r.si, _n3s.cfe, _n3s.cfs, _n3s.si, _n3t.cfe, _n3t.cfs, _n3t.si, _n57.cfe, _n57.cfs, _n57.si, _n58.cfe, _n58.cfs, _n58.si, _n59.cfe, _n59.cfs, _n59.si, _n5a.cfe, _n5a.cfs, _n5a.si, segments_2]}, {indexVersion=1487826451314,generation=3,filelist=[_n3v.fdt, _n3v.fdx, _n3v.fnm, _n3v.nvd, _n3v.nvm, _n3v.si, _n3v_Lucene50_0.doc, _n3v_Lucene50_0.tim, _n3v_Lucene50_0.tip, _n57.cfe, _n57.cfs, _n57.si, _n58.cfe, _n58.cfs, _n58.si, _n59.cfe, _n59.cfs, _n59.si, _n5a.cfe, _n5a.cfs, _n5a.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1487826451314,generation=2,filelist=[_n27.fdt, _n27.fdx, _n27.fnm, _n27.nvd, _n27.nvm, _n27.si, _n27_Lucene50_0.doc, _n27_Lucene50_0.tim, _n27_Lucene50_0.tip, _n3p.cfe, _n3p.cfs, _n3p.si, _n3q.cfe, _n3q.cfs, _n3q.si, _n3r.cfe, _n3r.cfs, _n3r.si, _n3s.cfe, _n3s.cfs, _n3s.si, _n3t.cfe, _n3t.cfs, _n3t.si, _n57.cfe, _n57.cfs, _n57.si, _n58.cfe, _n58.cfs, _n58.si, _n59.cfe, _n59.cfs, _n59.si, _n5a.cfe, _n5a.cfs, _n5a.si, segments_2]}]> but was:<[{indexVersion=1487826451314,generation=2,filelist=[_n27.fdt, _n27.fdx, _n27.fnm, _n27.nvd, _n27.nvm, _n27.si, _n27_Lucene50_0.doc, _n27_Lucene50_0.tim, _n27_Lucene50_0.tip, _n3p.cfe, _n3p.cfs, _n3p.si, _n3q.cfe, _n3q.cfs, _n3q.si, _n3r.cfe, _n3r.cfs, _n3r.si, _n3s.cfe, _n3s.cfs, _n3s.si, _n3t.cfe, _n3t.cfs, _n3t.si, _n57.cfe, _n57.cfs, _n57.si, _n58.cfe, _n58.cfs, _n58.si, _n59.cfe, _n59.cfs, _n59.si, _n5a.cfe, _n5a.cfs, _n5a.si, segments_2]}, {indexVersion=1487826451314,generation=3,filelist=[_n3v.fdt, _n3v.fdx, _n3v.fnm, _n3v.nvd, _n3v.nvm, _n3v.si, _n3v_Lucene50_0.doc, _n3v_Lucene50_0.tim, _n3v_Lucene50_0.tip, _n57.cfe, _n57.cfs, _n57.si, _n58.cfe, _n58.cfs, _n58.si, _n59.cfe, _n59.cfs, _n59.si, _n5a.cfe, _n5a.cfs, _n5a.si, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([BFA67DA57E6DA6E2:9A7166950E25A8E1]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1286)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12835 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/init-core-data-001
   [junit4]   2> 1696113 INFO  (SUITE-TestReplicationHandler-seed#[BFA67DA57E6DA6E2]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1696114 INFO  (SUITE-TestReplicationHandler-seed#[BFA67DA57E6DA6E2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=None)
   [junit4]   2> 1696115 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestReplicateAfterCoreReload
   [junit4]   2> 1696116 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/collection1
   [junit4]   2> 1696119 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1696120 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@228a7bf0{/solr,null,AVAILABLE}
   [junit4]   2> 1696122 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@bf6ab93{HTTP/1.1,[http/1.1]}{127.0.0.1:?????}
   [junit4]   2> 1696122 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.Server Started @???????ms
   [junit4]   2> 1696122 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=36833}
   [junit4]   2> 1696122 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1696122 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1696122 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1696122 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1696122 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-23T05:00:39.397Z
   [junit4]   2> 1696123 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/solr.xml
   [junit4]   2> 1696148 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/.
   [junit4]   2> 1696148 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1696156 WARN  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1696156 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1696162 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1696203 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1696204 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/./collection1
   [junit4]   2> 1696204 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1696205 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1696234 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1696234 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1696234 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=15.546875, floorSegmentMB=1.3671875, forceMergeDeletesPctAllowed=24.158308311862957, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7527091706087199
   [junit4]   2> 1696235 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@21fee2cf[collection1] main]
   [junit4]   2> 1696235 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/collection1/conf
   [junit4]   2> 1696236 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1696236 INFO  (coreLoadExecutor-5411-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1696236 INFO  (searcherExecutor-5412-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@21fee2cf[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1696604 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/collection1
   [junit4]   2> 1696606 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1696607 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41241c89{/solr,null,AVAILABLE}
   [junit4]   2> 1696607 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@2cc5952a{HTTP/1.1,[http/1.1]}{127.0.0.1:?????}
   [junit4]   2> 1696607 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.Server Started @???????ms
   [junit4]   2> 1696607 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=58163}
   [junit4]   2> 1696607 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1696607 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1696608 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1696608 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1696608 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-23T05:00:39.883Z
   [junit4]   2> 1696608 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/solr.xml
   [junit4]   2> 1696669 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/.
   [junit4]   2> 1696669 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1696678 WARN  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1696678 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1696684 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1696686 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1696687 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/./collection1
   [junit4]   2> 1696687 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1696689 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1696717 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1696717 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1696717 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=15.546875, floorSegmentMB=1.3671875, forceMergeDeletesPctAllowed=24.158308311862957, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7527091706087199
   [junit4]   2> 1696722 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@17d069f0[collection1] main]
   [junit4]   2> 1696723 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-002/collection1/conf
   [junit4]   2> 1696724 INFO  (coreLoadExecutor-5421-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1696725 INFO  (searcherExecutor-5422-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@17d069f0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1697093 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@2cc5952a{HTTP/1.1,[http/1.1]}{127.0.0.1:?}
   [junit4]   2> 1697093 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=92508648
   [junit4]   2> 1697094 INFO  (coreCloseExecutor-5427-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@27762ba9
   [junit4]   2> 1697101 INFO  (coreCloseExecutor-5427-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1697101 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1697102 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@41241c89{/solr,null,UNAVAILABLE}
   [junit4]   2> 1697103 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@bf6ab93{HTTP/1.1,[http/1.1]}{127.0.0.1:?}
   [junit4]   2> 1697103 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=679435942
   [junit4]   2> 1697103 INFO  (coreCloseExecutor-5429-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2b290f36
   [junit4]   2> 1697108 INFO  (coreCloseExecutor-5429-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 1697109 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1697115 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@228a7bf0{/solr,null,UNAVAILABLE}
   [junit4]   2> 1697116 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1697117 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@59d3394e{/solr,null,AVAILABLE}
   [junit4]   2> 1697117 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7cfaf8fe{HTTP/1.1,[http/1.1]}{127.0.0.1:?????}
   [junit4]   2> 1697117 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.e.j.s.Server Started @???????ms
   [junit4]   2> 1697117 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=43024}
   [junit4]   2> 1697118 ERROR (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1697118 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 1697118 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in standalone mode on port null
   [junit4]   2> 1697118 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1697118 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-23T05:00:40.393Z
   [junit4]   2> 1697118 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/solr.xml
   [junit4]   2> 1697150 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/.
   [junit4]   2> 1697150 INFO  (TEST-TestReplicationHandler.doTestReplicateAfterCoreReload-seed#[BFA67DA57E6DA6E2]) [    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1697173 WARN  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1697173 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1697214 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1697216 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
   [junit4]   2> 1697218 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/./collection1
   [junit4]   2> 1697218 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1697219 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=16, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1697244 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1697244 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1697244 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=15.546875, floorSegmentMB=1.3671875, forceMergeDeletesPctAllowed=24.158308311862957, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7527091706087199
   [junit4]   2> 1697253 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4992096c[collection1] main]
   [junit4]   2> 1697253 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler_BFA67DA57E6DA6E2-001/solr-instance-001/collection1/conf
   [junit4]   2> 1697253 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1697253 INFO  (coreLoadExecutor-5436-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1697258 INFO  (searcherExecutor-5437-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@4992096c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1697643 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 1697643 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 0
   [junit4]   2> 1697645 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 1
   [junit4]   2> 1697645 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 1697646 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 1697646 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 1697647 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[4]} 0 0
   [junit4]   2> 1697647 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[5]} 0 0
   [junit4]   2> 1697647 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 1697655 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[7]} 0 6
   [junit4]   2> 1697656 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[8]} 0 0
   [junit4]   2> 1697656 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[9]} 0 0
   [junit4]   2> 1697657 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[10]} 0 0
   [junit4]   2> 1697657 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[11]} 0 0
   [junit4]   2> 1697657 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[12]} 0 0
   [junit4]   2> 1697658 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[13]} 0 0
   [junit4]   2> 1697658 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[14]} 0 0
   [junit4]   2> 1697662 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[15]} 0 3
   [junit4]   2> 1697663 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 1697663 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[17]} 0 0
   [junit4]   2> 1697663 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[18]} 0 0
   [junit4]   2> 1697664 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[19]} 0 0
   [junit4]   2> 1697664 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[20]} 0 0
   [junit4]   2> 1697664 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[21]} 0 0
   [junit4]   2> 1697664 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[22]} 0 0
   [junit4]   2> 1697668 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[23]} 0 3
   [junit4]   2> 1697669 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[24]} 0 0
   [junit4]   2> 1697669 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[25]} 0 0
   [junit4]   2> 1697670 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[26]} 0 0
   [junit4]   2> 1697670 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[27]} 0 0
   [junit4]   2> 1697670 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[28]} 0 0
   [junit4]   2> 1697671 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[29]} 0 0
   [junit4]   2> 1697671 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[30]} 0 0
   [junit4]   2> 1697676 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[31]} 0 4
   [junit4]   2> 1697676 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[32]} 0 0
   [junit4]   2> 1697677 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[33]} 0 0
   [junit4]   2> 1697677 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[34]} 0 0
   [junit4]   2> 1697677 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[35]} 0 0
   [junit4]   2> 1697678 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[36]} 0 0
   [junit4]   2> 1697678 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[37]} 0 0
   [junit4]   2> 1697678 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[38]} 0 0
   [junit4]   2> 1697682 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[39]} 0 3
   [junit4]   2> 1697683 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[40]} 0 0
   [junit4]   2> 1697683 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[41]} 0 0
   [junit4]   2> 1697683 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[42]} 0 0
   [junit4]   2> 1697684 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[43]} 0 0
   [junit4]   2> 1697684 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[44]} 0 0
   [junit4]   2> 1697684 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[45]} 0 0
   [junit4]   2> 1697684 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[46]} 0 0
   [junit4]   2> 1697688 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[47]} 0 3
   [junit4]   2> 1697689 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[48]} 0 0
   [junit4]   2> 1697689 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[49]} 0 0
   [junit4]   2> 1697690 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[50]} 0 0
   [junit4]   2> 1697690 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[51]} 0 0
   [junit4]   2> 1697690 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[52]} 0 0
   [junit4]   2> 1697690 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[53]} 0 0
   [junit4]   2> 1697691 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[54]} 0 0
   [junit4]   2> 1697697 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[55]} 0 6
   [junit4]   2> 1697698 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[56]} 0 0
   [junit4]   2> 1697698 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[57]} 0 0
   [junit4]   2> 1697698 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[58]} 0 0
   [junit4]   2> 1697698 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[59]} 0 0
   [junit4]   2> 1697699 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[60]} 0 0
   [junit4]   2> 1697699 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[61]} 0 0
   [junit4]   2> 1697699 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[62]} 0 0
   [junit4]   2> 1697707 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[63]} 0 7
   [junit4]   2> 1697707 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[64]} 0 0
   [junit4]   2> 1697708 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[65]} 0 0
   [junit4]   2> 1697708 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[66]} 0 0
   [junit4]   2> 1697708 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[67]} 0 0
   [junit4]   2> 1697708 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[68]} 0 0
   [junit4]   2> 1697709 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[69]} 0 0
   [junit4]   2> 1697709 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[70]} 0 0
   [junit4]   2> 1697713 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[71]} 0 4
   [junit4]   2> 1697714 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[72]} 0 0
   [junit4]   2> 1697714 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[73]} 0 0
   [junit4]   2> 1697714 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[74]} 0 0
   [junit4]   2> 1697715 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[75]} 0 0
   [junit4]   2> 1697715 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[76]} 0 0
   [junit4]   2> 1697715 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[77]} 0 0
   [junit4]   2> 1697715 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[78]} 0 0
   [junit4]   2> 1697719 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[79]} 0 3
   [junit4]   2> 1697720 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[80]} 0 0
   [junit4]   2> 1697720 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[81]} 0 0
   [junit4]   2> 1697720 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[82]} 0 0
   [junit4]   2> 1697721 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[83]} 0 0
   [junit4]   2> 1697721 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[84]} 0 0
   [junit4]   2> 1697721 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[85]} 0 0
   [junit4]   2> 1697721 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[86]} 0 0
   [junit4]   2> 1697725 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[87]} 0 3
   [junit4]   2> 1697726 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[88]} 0 0
   [junit4]   2> 1697727 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[89]} 0 0
   [junit4]   2> 1697727 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[90]} 0 0
   [junit4]   2> 1697727 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[91]} 0 0
   [junit4]   2> 1697728 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[92]} 0 0
   [junit4]   2> 1697728 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[93]} 0 0
   [junit4]   2> 1697728 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[94]} 0 0
   [junit4]   2> 1697735 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[95]} 0 6
   [junit4]   2> 1697736 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[96]} 0 0
   [junit4]   2> 1697737 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[97]} 0 0
   [junit4]   2> 1697737 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[98]} 0 0
   [junit4]   2> 1697737 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[99]} 0 0
   [junit4]   2> 1697738 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 1697738 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 1697738 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 1697742 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[103]} 0 3
   [junit4]   2> 1697743 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 1697743 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 1697743 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 1697744 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 1697744 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 1697744 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 1697744 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 1697749 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[111]} 0 4
   [junit4]   2> 1697750 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 1697750 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 1697750 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 1697751 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 1697751 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 1697751 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 1697752 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 1697756 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[119]} 0 4
   [junit4]   2> 1697756 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 1697757 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 1697757 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 1697757 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 1697758 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 1697758 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 1697758 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 1697762 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[127]} 0 3
   [junit4]   2> 1697763 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 1697763 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 1697763 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 1697763 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 1697764 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[132]} 0 0
   [junit4]   2> 1697764 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 1697766 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[134]} 0 1
   [junit4]   2> 1697772 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[135]} 0 6
   [junit4]   2> 1697773 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 1697773 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 1697774 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 1697774 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[139]} 0 0
   [junit4]   2> 1697774 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 1697774 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 1697775 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 1697779 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[143]} 0 4
   [junit4]   2> 1697780 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 1697780 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 1697781 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 1697781 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 1697781 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 1697782 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 1697782 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[150]} 0 0
   [junit4]   2> 1697785 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[151]} 0 3
   [junit4]   2> 1697786 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[152]} 0 0
   [junit4]   2> 1697786 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[153]} 0 0
   [junit4]   2> 1697787 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[154]} 0 0
   [junit4]   2> 1697787 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[155]} 0 0
   [junit4]   2> 1697787 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[156]} 0 0
   [junit4]   2> 1697788 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[157]} 0 0
   [junit4]   2> 1697788 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[158]} 0 0
   [junit4]   2> 1697791 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[159]} 0 3
   [junit4]   2> 1697792 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[160]} 0 0
   [junit4]   2> 1697793 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[161]} 0 0
   [junit4]   2> 1697793 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[162]} 0 0
   [junit4]   2> 1697793 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[163]} 0 0
   [junit4]   2> 1697794 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[164]} 0 0
   [junit4]   2> 1697794 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[165]} 0 0
   [junit4]   2> 1697794 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[166]} 0 0
   [junit4]   2> 1697798 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[167]} 0 3
   [junit4]   2> 1697799 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[168]} 0 0
   [junit4]   2> 1697799 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[169]} 0 0
   [junit4]   2> 1697800 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[170]} 0 0
   [junit4]   2> 1697800 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[171]} 0 0
   [junit4]   2> 1697800 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[172]} 0 0
   [junit4]   2> 1697801 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[173]} 0 0
   [junit4]   2> 1697801 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[174]} 0 0
   [junit4]   2> 1697807 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[175]} 0 5
   [junit4]   2> 1697807 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[176]} 0 0
   [junit4]   2> 1697808 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[177]} 0 0
   [junit4]   2> 1697808 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[178]} 0 0
   [junit4]   2> 1697808 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[179]} 0 0
   [junit4]   2> 1697809 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[180]} 0 0
   [junit4]   2> 1697809 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[181]} 0 0
   [junit4]   2> 1697809 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[182]} 0 0
   [junit4]   2> 1697813 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[183]} 0 3
   [junit4]   2> 1697814 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[184]} 0 0
   [junit4]   2> 1697814 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[185]} 0 0
   [junit4]   2> 1697814 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[186]} 0 0
   [junit4]   2> 1697815 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[187]} 0 0
   [junit4]   2> 1697815 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[188]} 0 0
   [junit4]   2> 1697815 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[189]} 0 0
   [junit4]   2> 1697816 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[190]} 0 0
   [junit4]   2> 1697819 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[191]} 0 3
   [junit4]   2> 1697820 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[192]} 0 0
   [junit4]   2> 1697820 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[193]} 0 0
   [junit4]   2> 1697821 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[194]} 0 0
   [junit4]   2> 1697821 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[195]} 0 0
   [junit4]   2> 1697821 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[196]} 0 0
   [junit4]   2> 1697822 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[197]} 0 0
   [junit4]   2> 1697822 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[198]} 0 0
   [junit4]   2> 1697827 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[199]} 0 4
   [junit4]   2> 1697828 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[200]} 0 0
   [junit4]   2> 1697828 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[201]} 0 0
   [junit4]   2> 1697828 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[202]} 0 0
   [junit4]   2> 1697829 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[203]} 0 0
   [junit4]   2> 1697829 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[204]} 0 0
   [junit4]   2> 1697829 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[205]} 0 0
   [junit4]   2> 1697830 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[206]} 0 0
   [junit4]   2> 1697834 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[207]} 0 3
   [junit4]   2> 1697835 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[208]} 0 0
   [junit4]   2> 1697835 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[209]} 0 0
   [junit4]   2> 1697835 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[210]} 0 0
   [junit4]   2> 1697836 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[211]} 0 0
   [junit4]   2> 1697836 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[212]} 0 0
   [junit4]   2> 1697836 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[213]} 0 0
   [junit4]   2> 1697837 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[214]} 0 0
   [junit4]   2> 1697843 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[215]} 0 6
   [junit4]   2> 1697844 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[216]} 0 0
   [junit4]   2> 1697844 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[217]} 0 0
   [junit4]   2> 1697844 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[218]} 0 0
   [junit4]   2> 1697845 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[219]} 0 0
   [junit4]   2> 1697845 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[220]} 0 0
   [junit4]   2> 1697845 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[221]} 0 0
   [junit4]   2> 1697845 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[222]} 0 0
   [junit4]   2> 1697849 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[223]} 0 3
   [junit4]   2> 1697850 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[224]} 0 0
   [junit4]   2> 1697850 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[225]} 0 0
   [junit4]   2> 1697850 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[226]} 0 0
   [junit4]   2> 1697851 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[227]} 0 0
   [junit4]   2> 1697851 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[228]} 0 0
   [junit4]   2> 1697851 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[229]} 0 0
   [junit4]   2> 1697851 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[230]} 0 0
   [junit4]   2> 1697855 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[231]} 0 3
   [junit4]   2> 1697856 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[232]} 0 0
   [junit4]   2> 1697856 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[233]} 0 0
   [junit4]   2> 1697856 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[234]} 0 0
   [junit4]   2> 1697857 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[235]} 0 0
   [junit4]   2> 1697857 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[236]} 0 0
   [junit4]   2> 1697857 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[237]} 0 0
   [junit4]   2> 1697858 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[238]} 0 0
   [junit4]   2> 1697862 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[239]} 0 4
   [junit4]   2> 1697863 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[240]} 0 0
   [junit4]   2> 1697863 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[241]} 0 0
   [junit4]   2> 1697863 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[242]} 0 0
   [junit4]   2> 1697864 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[243]} 0 0
   [junit4]   2> 1697864 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[244]} 0 0
   [junit4]   2> 1697864 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[245]} 0 0
   [junit4]   2> 1697865 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[246]} 0 0
   [junit4]   2> 1697868 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[247]} 0 3
   [junit4]   2> 1697869 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[248]} 0 0
   [junit4]   2> 1697869 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[249]} 0 0
   [junit4]   2> 1697870 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[250]} 0 0
   [junit4]   2> 1697870 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[251]} 0 0
   [junit4]   2> 1697870 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[252]} 0 0
   [junit4]   2> 1697870 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[253]} 0 0
   [junit4]   2> 1697871 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[254]} 0 0
   [junit4]   2> 1697876 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[255]} 0 3
   [junit4]   2> 1697877 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[256]} 0 0
   [junit4]   2> 1697877 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[257]} 0 0
   [junit4]   2> 1697877 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[258]} 0 0
   [junit4]   2> 1697878 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[259]} 0 0
   [junit4]   2> 1697878 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[260]} 0 0
   [junit4]   2> 1697878 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[261]} 0 0
   [junit4]   2> 1697878 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[262]} 0 0
   [junit4]   2> 1697884 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[263]} 0 5
   [junit4]   2> 1697885 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[264]} 0 0
   [junit4]   2> 1697885 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[265]} 0 0
   [junit4]   2> 1697885 INFO  (qtp734600321-18283) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[266]} 0 0
   [junit4]   2> 1697886 INFO  (qtp734600321-18287) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[267]} 0 0
   [junit4]   2> 1697886 INFO  (qtp734600321-18286) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[268]} 0 0
   [junit4]   2> 1697886 INFO  (qtp734600321-18290) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[269]} 0 0
   [junit4]   2> 1697887 INFO  (qtp734600321-18288) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[270]} 0 0
   [junit4]   2> 1697891 INFO  (qtp734600321-18289) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[271]} 0 3
   [junit4]   2> 1697891 INFO  (q

[...truncated too long message...]

mission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactoryTest.teardownClass(HdfsDirectoryFactoryTest.java:61)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> 	... 53 more
   [junit4]   2> 3492825 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[BFA67DA57E6DA6E2]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.core.HdfsDirectoryFactoryTest_BFA67DA57E6DA6E2-001
   [junit4]   2> Feb 23, 2017 5:30:36 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=869, maxMBSortInHeap=7.585630510346611, sim=RandomSimilarity(queryNorm=true): {}, locale=en-AU, timezone=Europe/Volgograd
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=353456016,total=509083648
   [junit4]   2> NOTE: All tests run in this JVM: [TestRuleBasedAuthorizationPlugin, RestartWhileUpdatingTest, TestUniqueKeyFieldResource, ZkSolrClientTest, TestFastLRUCache, TestLeaderInitiatedRecoveryThread, TestDistributedGrouping, ParsingFieldUpdateProcessorsTest, BlockJoinFacetDistribTest, SimplePostToolTest, JsonLoaderTest, DistributedExpandComponentTest, ConcurrentDeleteAndCreateCollectionTest, IndexSchemaRuntimeFieldTest, TestTrieFacet, ReplicaListTransformerTest, TestXIncludeConfig, TestChildDocTransformer, TemplateUpdateProcessorTest, TestCloudSchemaless, TestRebalanceLeaders, ZkStateWriterTest, ConvertedLegacyTest, BigEndianAscendingWordSerializerTest, TestJsonFacetRefinement, DistribDocExpirationUpdateProcessorTest, TermVectorComponentTest, SuggestComponentContextFilterQueryTest, TestConfigSetsAPIZkFailure, AddBlockUpdateTest, TestStressRecovery, TestDistribIDF, UpdateRequestProcessorFactoryTest, SpellCheckComponentTest, CollectionTooManyReplicasTest, SuggesterFSTTest, CreateCollectionCleanupTest, TestPostingsSolrHighlighter, CdcrReplicationDistributedZkTest, TestSearchPerf, TestInfoStreamLogging, SolrCmdDistributorTest, TestExpandComponent, BufferStoreTest, AsyncCallRequestStatusResponseTest, TermsComponentTest, SolrInfoMBeanTest, DOMUtilTest, SolrCloudExampleTest, SaslZkACLProviderTest, TestSolrCoreSnapshots, DistributedTermsComponentTest, BasicFunctionalityTest, TestSchemaManager, TestFieldSortValues, CloneFieldUpdateProcessorFactoryTest, TestConfigSetProperties, RemoteQueryErrorTest, DistributedVersionInfoTest, TestNumericTerms32, TestSimpleTrackingShardHandler, MigrateRouteKeyTest, JSONWriterTest, TestImpersonationWithHadoopAuth, TestHdfsCloudBackupRestore, TestBulkSchemaConcurrent, TestPerFieldSimilarity, TestReloadAndDeleteDocs, ClusterStateUpdateTest, AlternateDirectoryTest, ResponseHeaderTest, TestQueryUtils, VersionInfoTest, TestIntervalFaceting, TestHashPartitioner, TestFieldCacheVsDocValues, CursorMarkTest, OpenExchangeRatesOrgProviderTest, RulesTest, SortSpecParsingTest, SoftAutoCommitTest, TestRawResponseWriter, ClassificationUpdateProcessorFactoryTest, SpellingQueryConverterTest, TestStressInPlaceUpdates, TestMaxScoreQueryParser, TestCopyFieldCollectionResource, TestSchemaResource, TestJsonRequest, TestGraphMLResponseWriter, IndexBasedSpellCheckerTest, CdcrVersionReplicationTest, ManagedSchemaRoundRobinCloudTest, VMParamsZkACLAndCredentialsProvidersTest, TestElisionMultitermQuery, UUIDUpdateProcessorFallbackTest, TestSystemIdResolver, TestLRUStatsCache, TestWordDelimiterFilterFactory, DeleteLastCustomShardedReplicaTest, TestSQLHandlerNonCloud, TestCloudPseudoReturnFields, TestHighFrequencyDictionaryFactory, TestBM25SimilarityFactory, TestCustomDocTransformer, ZkCLITest, SortByFunctionTest, TestSweetSpotSimilarityFactory, CollectionsAPIDistributedZkTest, TestSlowCompositeReaderWrapper, TestCollectionAPI, TestSerializedLuceneMatchVersion, FileUtilsTest, LeaderElectionIntegrationTest, ShardRoutingTest, TestRandomFaceting, ShardRoutingCustomTest, TestDistributedSearch, ZkControllerTest, TestJoin, TestReload, TestStressVersions, HardAutoCommitTest, TestRangeQuery, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestTolerantSearch, TestPathTrie, TestEmbeddedSolrServerConstructors, ConnectionReuseTest, ActionThrottleTest, AliasIntegrationTest, BaseCdcrDistributedZkTest, CdcrBootstrapTest, CdcrRequestHandlerTest, DeleteNodeTest, DeleteStatusTest, HttpPartitionTest, RollingRestartTest, SharedFSAutoReplicaFailoverUtilsTest, SolrCLIZkUtilsTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudPivotFacet, TestConfigSetsAPI, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestExclusionRuleCollectionAccess, TestMiniSolrCloudCluster, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestStressLiveNodes, HdfsChaosMonkeyNothingIsSafeTest, HdfsNNFailoverTest, HdfsRestartWhileUpdatingTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest, CoreSorterTest, DirectoryFactoryTest, ExitableDirectoryReaderTest, HdfsDirectoryFactoryTest]
   [junit4] Completed [551/693 (2!)] on J1 in 12.73s, 3 tests, 1 error <<< FAILURES!

[...truncated 63048 lines...]



[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1251 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1251/

5 tests failed.
FAILED:  org.apache.lucene.search.TestDoubleRangeFieldQueries.testRandomMedium

Error Message:
wrong hit (first of possibly more):  FAIL (iter 75): id=52600 should not match but did  queryRange=Box(-Infinity TO Infinity)  box=Box(-3.7647432892534716E307 TO 5.309615037404456E307)  queryType=CROSSES  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 75): id=52600 should not match but did
 queryRange=Box(-Infinity TO Infinity)
 box=Box(-3.7647432892534716E307 TO 5.309615037404456E307)
 queryType=CROSSES
 deleted?=false
	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.TestFloatRangeFieldQueries.testRandomMedium

Error Message:
wrong hit (first of possibly more):  FAIL (iter 75): id=52600 should not match but did  queryRange=Box(-Infinity TO Infinity)  box=Box(-1.3110648E38 TO 1.4387144E38)  queryType=CROSSES  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 75): id=52600 should not match but did
 queryRange=Box(-Infinity TO Infinity)
 box=Box(-1.3110648E38 TO 1.4387144E38)
 queryType=CROSSES
 deleted?=false
	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.TestIntRangeFieldQueries.testRandomMedium

Error Message:
wrong hit (first of possibly more):  FAIL (iter 75): id=52600 should not match but did  queryRange=Box(-2147483648 TO 2147483647)  box=Box(-2059116195 TO -410600835)  queryType=CROSSES  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 75): id=52600 should not match but did
 queryRange=Box(-2147483648 TO 2147483647)
 box=Box(-2059116195 TO -410600835)
 queryType=CROSSES
 deleted?=false
	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.lucene.search.TestLongRangeFieldQueries.testRandomMedium

Error Message:
wrong hit (first of possibly more):  FAIL (iter 75): id=52600 should not match but did  queryRange=Box(-9223372036854775808 TO 9223372036854775807)  box=Box(2938302894405237373 TO 3114577670663594333)  queryType=CROSSES  deleted?=false

Stack Trace:
java.lang.AssertionError: wrong hit (first of possibly more):

FAIL (iter 75): id=52600 should not match but did
 queryRange=Box(-9223372036854775808 TO 9223372036854775807)
 box=Box(2938302894405237373 TO 3114577670663594333)
 queryType=CROSSES
 deleted?=false
	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Could not find collection:delete_data_dir

Stack Trace:
java.lang.AssertionError: Could not find collection:delete_data_dir
	at __randomizedtesting.SeedInfo.seed([80E1D51B68C41621:8B5EAC1C6387BD9]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:865)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:159)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9086 lines...]
   [junit4] Suite: org.apache.lucene.search.TestLongRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLongRangeFieldQueries -Dtests.method=testRandomMedium -Dtests.seed=372308A94C0C5F3B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=be-BY -Dtests.timezone=America/Indiana/Winamac -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 7.53s J2 | TestLongRangeFieldQueries.testRandomMedium <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 75): id=52600 should not match but did
   [junit4]    >  queryRange=Box(-9223372036854775808 TO 9223372036854775807)
   [junit4]    >  box=Box(2938302894405237373 TO 3114577670663594333)
   [junit4]    >  queryType=CROSSES
   [junit4]    >  deleted?=false
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/sandbox/test/J2/temp/lucene.search.TestLongRangeFieldQueries_372308A94C0C5F3B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=LuceneFixedGap)}, docValues:{id=DocValuesFormat(name=Direct), longRangeField=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=41, maxMBSortInHeap=5.573918837864136, sim=RandomSimilarity(queryNorm=true): {}, locale=be-BY, timezone=America/Indiana/Winamac
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=136358544,total=484966400
   [junit4]   2> NOTE: All tests run in this JVM: [TestLongRangeFieldQueries]
   [junit4] Completed [6/21 (1!)] on J2 in 157.20s, 5 tests, 1 failure <<< FAILURES!

[...truncated 32 lines...]
   [junit4] Suite: org.apache.lucene.search.TestIntRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIntRangeFieldQueries -Dtests.method=testRandomMedium -Dtests.seed=372308A94C0C5F3B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=en-ZA -Dtests.timezone=Europe/Helsinki -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 5.30s J2 | TestIntRangeFieldQueries.testRandomMedium <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 75): id=52600 should not match but did
   [junit4]    >  queryRange=Box(-2147483648 TO 2147483647)
   [junit4]    >  box=Box(-2059116195 TO -410600835)
   [junit4]    >  queryType=CROSSES
   [junit4]    >  deleted?=false
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/sandbox/test/J2/temp/lucene.search.TestIntRangeFieldQueries_372308A94C0C5F3B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{id=DocValuesFormat(name=Memory), intRangeField=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1034, maxMBSortInHeap=7.960284461233806, sim=RandomSimilarity(queryNorm=true): {}, locale=en-ZA, timezone=Europe/Helsinki
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=254900272,total=515375104
   [junit4]   2> NOTE: All tests run in this JVM: [TestLongRangeFieldQueries, TestNearest, TestIntRangeFieldQueries]
   [junit4] Completed [16/21 (2!)] on J2 in 122.33s, 5 tests, 1 failure <<< FAILURES!

[...truncated 11 lines...]
   [junit4] Suite: org.apache.lucene.search.TestDoubleRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDoubleRangeFieldQueries -Dtests.method=testRandomMedium -Dtests.seed=372308A94C0C5F3B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=mk-MK -Dtests.timezone=Japan -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 6.37s J0 | TestDoubleRangeFieldQueries.testRandomMedium <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 75): id=52600 should not match but did
   [junit4]    >  queryRange=Box(-Infinity TO Infinity)
   [junit4]    >  box=Box(-3.7647432892534716E307 TO 5.309615037404456E307)
   [junit4]    >  queryType=CROSSES
   [junit4]    >  deleted?=false
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/sandbox/test/J0/temp/lucene.search.TestDoubleRangeFieldQueries_372308A94C0C5F3B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{doubleRangeField=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=130, maxMBSortInHeap=7.386413603500476, sim=RandomSimilarity(queryNorm=false): {}, locale=mk-MK, timezone=Japan
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=197310064,total=495452160
   [junit4]   2> NOTE: All tests run in this JVM: [FuzzyLikeThisQueryTest, TestLatLonPoint, TestHalfFloatPoint, TestLatLonDocValuesQueries, TestIDVersionPostingsFormat, TestDoubleRangeField, TestInetAddressPoint, TestLatLonDocValuesField, TestLatLonPointDistanceSort, TestPayloadSpanUtil, TestDocValuesNumbersQuery, TestDocValuesTermsQuery, TestDoubleRangeFieldQueries]
   [junit4] Completed [20/21 (3!)] on J0 in 142.37s, 5 tests, 1 failure <<< FAILURES!

[...truncated 2 lines...]
   [junit4] Suite: org.apache.lucene.search.TestFloatRangeFieldQueries
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFloatRangeFieldQueries -Dtests.method=testRandomMedium -Dtests.seed=372308A94C0C5F3B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=de-CH -Dtests.timezone=SystemV/EST5EDT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 3.39s J1 | TestFloatRangeFieldQueries.testRandomMedium <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit (first of possibly more):
   [junit4]    > FAIL (iter 75): id=52600 should not match but did
   [junit4]    >  queryRange=Box(-Infinity TO Infinity)
   [junit4]    >  box=Box(-1.3110648E38 TO 1.4387144E38)
   [junit4]    >  queryType=CROSSES
   [junit4]    >  deleted?=false
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([372308A94C0C5F3B:8AFD3F010D693C5D]:0)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.verify(BaseRangeFieldQueryTestCase.java:287)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.doTestRandom(BaseRangeFieldQueryTestCase.java:158)
   [junit4]    > 	at org.apache.lucene.search.BaseRangeFieldQueryTestCase.testRandomMedium(BaseRangeFieldQueryTestCase.java:68)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/sandbox/test/J1/temp/lucene.search.TestFloatRangeFieldQueries_372308A94C0C5F3B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=LuceneFixedGap)}, docValues:{floatRangeField=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=1962, maxMBSortInHeap=7.732684305995514, sim=RandomSimilarity(queryNorm=false): {}, locale=de-CH, timezone=SystemV/EST5EDT
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=168969784,total=381157376
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldCacheTermsFilter, TestLatLonPointQueries, TestBigIntegerPoint, TestFloatRangeFieldQueries]
   [junit4] Completed [21/21 (4!)] on J1 in 80.45s, 5 tests, 1 failure <<< FAILURES!

[...truncated 3957 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/init-core-data-001
   [junit4]   2> 3328678 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 3328679 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 3328679 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 3328703 WARN  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 3328707 WARN  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3328708 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3328718 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_48380_hdfs____hibksx/webapp
   [junit4]   2> 3329063 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:48380
   [junit4]   2> 3329111 WARN  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3329111 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3329120 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_44732_datanode____.myhoeb/webapp
   [junit4]   2> 3329473 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44732
   [junit4]   2> 3329519 WARN  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 3329520 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 3329531 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_46872_datanode____mo9whk/webapp
   [junit4]   2> 3329586 INFO  (IPC Server handler 3 on 51168) [    ] BlockStateChange BLOCK* processReport: from storage DS-01875f85-4fc2-456e-b018-57fdc8b4dc84 node DatanodeRegistration(127.0.0.1:45039, datanodeUuid=0ebd197a-3436-485e-8c77-c9ce55eae7fa, infoPort=60307, infoSecurePort=0, ipcPort=33421, storageInfo=lv=-56;cid=testClusterID;nsid=527590998;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3329586 INFO  (IPC Server handler 3 on 51168) [    ] BlockStateChange BLOCK* processReport: from storage DS-bc3f4bdd-1b22-40ed-852c-19b6309b9ce8 node DatanodeRegistration(127.0.0.1:45039, datanodeUuid=0ebd197a-3436-485e-8c77-c9ce55eae7fa, infoPort=60307, infoSecurePort=0, ipcPort=33421, storageInfo=lv=-56;cid=testClusterID;nsid=527590998;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3329881 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46872
   [junit4]   2> 3329973 INFO  (IPC Server handler 8 on 51168) [    ] BlockStateChange BLOCK* processReport: from storage DS-0436500d-4bbd-4ac1-a58c-843268b5aa15 node DatanodeRegistration(127.0.0.1:43592, datanodeUuid=e4aaf44b-8dbb-4805-8d3c-892e0e64d169, infoPort=41560, infoSecurePort=0, ipcPort=41176, storageInfo=lv=-56;cid=testClusterID;nsid=527590998;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 3329973 INFO  (IPC Server handler 8 on 51168) [    ] BlockStateChange BLOCK* processReport: from storage DS-7bb2704f-5e0f-46b5-bcbd-1faf0dff9076 node DatanodeRegistration(127.0.0.1:43592, datanodeUuid=e4aaf44b-8dbb-4805-8d3c-892e0e64d169, infoPort=41560, infoSecurePort=0, ipcPort=41176, storageInfo=lv=-56;cid=testClusterID;nsid=527590998;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 3330058 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3330058 INFO  (Thread-22146) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3330058 INFO  (Thread-22146) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3330158 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.ZkTestServer start zk server on port:54454
   [junit4]   2> 3330171 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3330173 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3330174 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3330175 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3330176 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3330177 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3330177 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3330178 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3330179 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3330180 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3330181 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3330253 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/control-001/cores/collection1
   [junit4]   2> 3330254 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3330255 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@62e2842d{/,null,AVAILABLE}
   [junit4]   2> 3330255 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@700dc10f{HTTP/1.1,[http/1.1]}{127.0.0.1:33699}
   [junit4]   2> 3330256 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.Server Started @3333484ms
   [junit4]   2> 3330256 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:51168/hdfs__localhost_51168__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001_tempDir-002_control_data, hostContext=/, hostPort=33699, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/control-001/cores}
   [junit4]   2> 3330256 ERROR (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3330256 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.0.0
   [junit4]   2> 3330256 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3330256 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3330256 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-01T13:20:13.031Z
   [junit4]   2> 3330262 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3330262 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/control-001/solr.xml
   [junit4]   2> 3330268 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54454/solr
   [junit4]   2> 3330282 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:33699_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3330283 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:33699_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33699_
   [junit4]   2> 3330284 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:33699_    ] o.a.s.c.Overseer Overseer (id=97542105519489028-127.0.0.1:33699_-n_0000000000) starting
   [junit4]   2> 3330288 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:33699_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33699_
   [junit4]   2> 3330290 INFO  (zkCallback-3952-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3330351 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:33699_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/control-001/cores
   [junit4]   2> 3330351 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:33699_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3330357 INFO  (OverseerStateUpdate-97542105519489028-127.0.0.1:33699_-n_0000000000) [n:127.0.0.1:33699_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3330372 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3330382 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3330458 WARN  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 3330471 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3330478 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 3330479 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:51168/solr_hdfs_home
   [junit4]   2> 3330479 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3330479 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/control-001/cores/collection1], dataDir=[hdfs://localhost:51168/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 3330479 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43495c25
   [junit4]   2> 3330479 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:51168/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 3330485 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3330485 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3330485 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3330490 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3330490 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:51168/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 3330501 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:51168/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 3330506 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3330506 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3330506 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3330510 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3330510 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=87.0546875, floorSegmentMB=1.5966796875, forceMergeDeletesPctAllowed=13.28550622943114, segmentsPerTier=3.0, maxCFSSegmentSizeMB=2.107421875, noCFSRatio=1.0
   [junit4]   2> 3330524 INFO  (IPC Server handler 1 on 51168) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43592 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bc3f4bdd-1b22-40ed-852c-19b6309b9ce8:NORMAL:127.0.0.1:45039|RBW], ReplicaUC[[DISK]DS-7bb2704f-5e0f-46b5-bcbd-1faf0dff9076:NORMAL:127.0.0.1:43592|FINALIZED]]} size 0
   [junit4]   2> 3330525 INFO  (IPC Server handler 2 on 51168) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45039 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-bc3f4bdd-1b22-40ed-852c-19b6309b9ce8:NORMAL:127.0.0.1:45039|RBW], ReplicaUC[[DISK]DS-7bb2704f-5e0f-46b5-bcbd-1faf0dff9076:NORMAL:127.0.0.1:43592|FINALIZED]]} size 0
   [junit4]   2> 3330530 WARN  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3330562 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3330562 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3330562 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3330569 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3330569 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3330570 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=57.9267578125, floorSegmentMB=1.9970703125, forceMergeDeletesPctAllowed=20.576253553614062, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8680518771599125
   [junit4]   2> 3330573 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6c49d50f[collection1] main]
   [junit4]   2> 3330573 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3330574 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3330574 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3330575 INFO  (searcherExecutor-7840-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6c49d50f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3330576 INFO  (coreLoadExecutor-7839-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560673688853938176
   [junit4]   2> 3330582 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3330582 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3330582 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33699/collection1/
   [junit4]   2> 3330582 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3330582 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:33699/collection1/ has no replicas
   [junit4]   2> 3330582 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3330586 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33699/collection1/ shard1
   [junit4]   2> 3330738 INFO  (coreZkRegister-7832-thread-1-processing-n:127.0.0.1:33699_ x:collection1 c:control_collection) [n:127.0.0.1:33699_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3330792 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3330793 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54454/solr ready
   [junit4]   2> 3330793 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3330793 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 3330864 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-1-001/cores/collection1
   [junit4]   2> 3330864 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-1-001
   [junit4]   2> 3330865 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3330866 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b80fbb4{/,null,AVAILABLE}
   [junit4]   2> 3330866 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@42711f54{HTTP/1.1,[http/1.1]}{127.0.0.1:48407}
   [junit4]   2> 3330866 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.Server Started @3334095ms
   [junit4]   2> 3330866 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:51168/hdfs__localhost_51168__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=48407, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-1-001/cores}
   [junit4]   2> 3330866 ERROR (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3330866 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.0.0
   [junit4]   2> 3330867 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3330867 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3330867 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-01T13:20:13.642Z
   [junit4]   2> 3330869 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3330869 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-1-001/solr.xml
   [junit4]   2> 3330874 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54454/solr
   [junit4]   2> 3330881 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3330882 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:48407_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3330885 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:48407_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48407_
   [junit4]   2> 3330886 INFO  (zkCallback-3956-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3330886 INFO  (zkCallback-3952-thread-3-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3330893 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3330946 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:48407_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-1-001/cores
   [junit4]   2> 3330946 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:48407_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3330950 INFO  (OverseerStateUpdate-97542105519489028-127.0.0.1:33699_-n_0000000000) [n:127.0.0.1:33699_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3331051 WARN  (OverseerStateUpdate-97542105519489028-127.0.0.1:33699_-n_0000000000) [n:127.0.0.1:33699_    ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
   [junit4]   2> 3331054 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3331055 INFO  (OverseerStateUpdate-97542105519489028-127.0.0.1:33699_-n_0000000000) [n:127.0.0.1:33699_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3331055 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3331958 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3331968 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3332080 WARN  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 3332082 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3332101 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 3332101 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:51168/solr_hdfs_home
   [junit4]   2> 3332101 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3332101 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-1-001/cores/collection1], dataDir=[hdfs://localhost:51168/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 3332102 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43495c25
   [junit4]   2> 3332102 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:51168/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 3332136 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3332136 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3332136 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3332139 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3332139 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:51168/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 3332153 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:51168/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 3332160 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3332160 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3332160 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3332163 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3332164 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=87.0546875, floorSegmentMB=1.5966796875, forceMergeDeletesPctAllowed=13.28550622943114, segmentsPerTier=3.0, maxCFSSegmentSizeMB=2.107421875, noCFSRatio=1.0
   [junit4]   2> 3332175 INFO  (IPC Server handler 8 on 51168) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43592 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-01875f85-4fc2-456e-b018-57fdc8b4dc84:NORMAL:127.0.0.1:45039|RBW], ReplicaUC[[DISK]DS-0436500d-4bbd-4ac1-a58c-843268b5aa15:NORMAL:127.0.0.1:43592|FINALIZED]]} size 0
   [junit4]   2> 3332176 INFO  (IPC Server handler 9 on 51168) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45039 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-01875f85-4fc2-456e-b018-57fdc8b4dc84:NORMAL:127.0.0.1:45039|RBW], ReplicaUC[[DISK]DS-0436500d-4bbd-4ac1-a58c-843268b5aa15:NORMAL:127.0.0.1:43592|FINALIZED]]} size 0
   [junit4]   2> 3332182 WARN  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 3332211 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 3332211 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3332211 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 3332219 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3332219 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3332221 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=57.9267578125, floorSegmentMB=1.9970703125, forceMergeDeletesPctAllowed=20.576253553614062, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8680518771599125
   [junit4]   2> 3332223 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@50f8f316[collection1] main]
   [junit4]   2> 3332224 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3332224 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3332224 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3332225 INFO  (searcherExecutor-7851-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@50f8f316[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3332227 INFO  (coreLoadExecutor-7850-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560673690585137152
   [junit4]   2> 3332232 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3332233 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3332233 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:48407/collection1/
   [junit4]   2> 3332233 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3332233 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:48407/collection1/ has no replicas
   [junit4]   2> 3332233 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 3332236 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:48407/collection1/ shard1
   [junit4]   2> 3332339 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 3332387 INFO  (coreZkRegister-7845-thread-1-processing-n:127.0.0.1:48407_ x:collection1 c:collection1) [n:127.0.0.1:48407_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3332406 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-2-001/cores/collection1
   [junit4]   2> 3332406 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-2-001
   [junit4]   2> 3332408 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 3332409 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3100d7d2{/,null,AVAILABLE}
   [junit4]   2> 3332409 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7cc9507d{HTTP/1.1,[http/1.1]}{127.0.0.1:56047}
   [junit4]   2> 3332409 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.e.j.s.Server Started @3335638ms
   [junit4]   2> 3332409 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:51168/hdfs__localhost_51168__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001_tempDir-002_jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=56047, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-2-001/cores}
   [junit4]   2> 3332409 ERROR (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3332409 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.0.0
   [junit4]   2> 3332410 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3332410 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 3332410 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-03-01T13:20:15.185Z
   [junit4]   2> 3332421 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3332421 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-2-001/solr.xml
   [junit4]   2> 3332429 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54454/solr
   [junit4]   2> 3332440 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:56047_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3332441 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:56047_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3332442 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:56047_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56047_
   [junit4]   2> 3332443 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3332444 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3332444 INFO  (zkCallback-3952-thread-1-processing-n:127.0.0.1:33699_) [n:127.0.0.1:33699_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3332443 INFO  (zkCallback-3956-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3332462 INFO  (zkCallback-3968-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 3332547 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:56047_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-2-001/cores
   [junit4]   2> 3332547 INFO  (TEST-StressHdfsTest.test-seed#[80E1D51B68C41621]) [n:127.0.0.1:56047_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 3332548 WARN  (OverseerStateUpdate-97542105519489028-127.0.0.1:33699_-n_0000000000) [n:127.0.0.1:33699_    ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
   [junit4]   2> 3332551 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3332551 INFO  (OverseerStateUpdate-97542105519489028-127.0.0.1:33699_-n_0000000000) [n:127.0.0.1:33699_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3332655 INFO  (zkCallback-3968-thread-2-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3332655 INFO  (zkCallback-3962-thread-1-processing-n:127.0.0.1:48407_) [n:127.0.0.1:48407_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 3333561 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3333571 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3333645 WARN  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 3333647 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 3333655 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 3333656 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:51168/solr_hdfs_home
   [junit4]   2> 3333656 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 3333656 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001/shard-2-001/cores/collection1], dataDir=[hdfs://localhost:51168/solr_hdfs_home/collection1/core_node2/data/]
   [junit4]   2> 3333656 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43495c25
   [junit4]   2> 3333656 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:51168/solr_hdfs_home/collection1/core_node2/data/snapshot_metadata
   [junit4]   2> 3333662 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 3333662 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 3333662 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 3333670 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 3333670 INFO  (coreLoadExecutor-7861-thread-1-processing-n:127.0.0.1:56047_) [n:127.0.0.1:56047_ c:collection1   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory fac

[...truncated too long message...]

BeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.StressHdfsTest.teardownClass(StressHdfsTest.java:77)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> 	... 53 more
   [junit4]   2> 3411798 INFO  (SUITE-StressHdfsTest-seed#[80E1D51B68C41621]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.StressHdfsTest_80E1D51B68C41621-001
   [junit4]   2> Mar 01, 2017 1:21:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=PostingsFormat(name=Memory), id=Lucene50(blocksize=128), text=Lucene50(blocksize=128), timestamp=PostingsFormat(name=LuceneFixedGap), txt_t=Lucene50(blocksize=128)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Direct), intDefault=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Direct), intDvoDefault=DocValuesFormat(name=Lucene70), range_facet_l=DocValuesFormat(name=Direct), timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=442, maxMBSortInHeap=5.886961882634159, sim=RandomSimilarity(queryNorm=false): {}, locale=ms-MY, timezone=America/Port-au-Prince
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=303460808,total=530055168
   [junit4]   2> NOTE: All tests run in this JVM: [NotRequiredUniqueKeyTest, SyncSliceTest, BlobRepositoryCloudTest, TestHighlightDedupGrouping, SearchHandlerTest, HLLUtilTest, V2ApiIntegrationTest, TestSortByMinMaxFunction, WrapperMergePolicyFactoryTest, TestRestoreCore, TestSolrCoreProperties, TestReplicationHandler, TestSolrDeletionPolicy2, BasicAuthIntegrationTest, SubstringBytesRefFilterTest, TestClusterStateMutator, ConjunctionSolrSpellCheckerTest, UpdateLogTest, DisMaxRequestHandlerTest, SecurityConfHandlerTest, XsltUpdateRequestHandlerTest, TestSolrQueryParserDefaultOperatorResource, TestFieldCacheSortRandom, SimpleCollectionCreateDeleteTest, HdfsDirectoryTest, ShardSplitTest, TestRawTransformer, BaseCdcrDistributedZkTest, TolerantUpdateProcessorTest, JvmMetricsTest, TestJoin, CdcrUpdateLogTest, TestFileDictionaryLookup, CloudMLTQParserTest, DistributedSpellCheckComponentTest, ConfigSetsAPITest, TestSSLRandomization, DeleteStatusTest, RequestLoggingTest, TestSolr4Spatial, TestCloudRecovery, TestSolr4Spatial2, TestRealTimeGet, DefaultValueUpdateProcessorTest, BasicZkTest, CursorPagingTest, TestSchemaSimilarityResource, TestPseudoReturnFields, MergeStrategyTest, TestInitParams, TestDFISimilarityFactory, HdfsBasicDistributedZk2Test, TestHdfsUpdateLog, AtomicUpdatesTest, TestRangeQuery, TestSizeLimitedDistributedMap, TestXmlQParser, TestShardHandlerFactory, SolrGraphiteReporterTest, TestSolrCloudWithSecureImpersonation, TestLocalFSCloudBackupRestore, TestSha256AuthenticationProvider, TestExportWriter, TestComplexPhraseQParserPlugin, MinimalSchemaTest, DistributedQueryElevationComponentTest, PreAnalyzedFieldManagedSchemaCloudTest, TestAtomicUpdateErrorCases, DocumentAnalysisRequestHandlerTest, LeaderInitiatedRecoveryOnCommitTest, StandardRequestHandlerTest, TestLegacyFieldCache, TestValueSourceCache, TestFieldCollectionResource, CdcrReplicationHandlerTest, SmileWriterTest, TestUninvertingReader, TestMissingGroups, TestLMJelinekMercerSimilarityFactory, PolyFieldTest, TestDownShardTolerantSearch, TestCrossCoreJoin, TestMacroExpander, TestExactSharedStatsCache, TestPathTrie, DocValuesMissingTest, TestCryptoKeys, TestCoreAdminApis, TestManagedResource, TestFilteredDocIdSet, SystemInfoHandlerTest, TestRandomFlRTGCloud, FacetPivotSmallTest, QueryElevationComponentTest, OverriddenZkACLAndCredentialsProvidersTest, MultiTermTest, TestAddFieldRealTimeGet, CollectionsAPIAsyncDistributedZkTest, ZkNodePropsTest, BigEndianAscendingWordDeserializerTest, CopyFieldTest, TestRTGBase, IgnoreCommitOptimizeUpdateProcessorFactoryTest, ClassificationUpdateProcessorTest, AliasIntegrationTest, TestJmxMonitoredMap, TestSolrConfigHandlerCloud, SolrXmlInZkTest, TestSearcherReuse, TestDocSet, TestSearchPerf, NumericFieldsTest, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, ResourceLoaderTest, TestFastOutputStream, URLClassifyProcessorTest, PluginInfoTest, DateMathParserTest, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, ClusterStateTest, TestUtils, CircularListTest, TestCursorMarkWithoutUniqueKey, TestSimpleTrackingShardHandler, ActionThrottleTest, CdcrRequestHandlerTest, DeleteInactiveReplicaTest, DeleteShardTest, DistribDocExpirationUpdateProcessorTest, ForceLeaderTest, SharedFSAutoReplicaFailoverUtilsTest, SolrCloudExampleTest, TestDistribDocBasedVersion, TestLeaderInitiatedRecoveryThread, TestMiniSolrCloudClusterSSL, TestReplicaProperties, TestStressInPlaceUpdates, TlogReplayBufferedWhileIndexingTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsChaosMonkeySafeLeaderTest, HdfsRecoverLeaseTest, HdfsSyncSliceTest, HdfsUnloadDistributedZkTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest]
   [junit4] Completed [544/693 (1!)] on J0 in 93.02s, 1 test, 1 failure <<< FAILURES!

[...truncated 63063 lines...]



Re: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1250 - Still unstable

Posted by Michael McCandless <lu...@mikemccandless.com>.
Wow :)  Nice catch!  Thanks Adrien.

Mike McCandless

http://blog.mikemccandless.com

On Tue, Feb 28, 2017 at 2:56 AM, Adrien Grand <jp...@gmail.com> wrote:

> I pushed a fix. This was an infinite loop due to the fact that the test
> tried to generate a number of unique terms that was greater than the
> vocabulary size.
>
> Le mar. 28 févr. 2017 à 11:43, Adrien Grand <jp...@gmail.com> a écrit :
>
>> This reproduces for me, I'll dig.
>>
>> Le mar. 28 févr. 2017 à 09:58, Apache Jenkins Server <
>> jenkins@builds.apache.org> a écrit :
>>
>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-maste
>> r/1250/
>>
>> 2 tests failed.
>> FAILED:  org.apache.lucene.search.TestFuzzyQuery.testRandom
>>
>> Error Message:
>> Test abandoned because suite timeout was reached.
>>
>> Stack Trace:
>> java.lang.Exception: Test abandoned because suite timeout was reached.
>>         at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>>
>>
>> FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestFuzzy
>> Query
>>
>> Error Message:
>> Suite timeout exceeded (>= 7200000 msec).
>>
>> Stack Trace:
>> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>>         at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 1695 lines...]
>>    [junit4] Suite: org.apache.lucene.search.TestFuzzyQuery
>>    [junit4]   2> Feb 27, 2017 10:38:13 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2
>> evaluate
>>    [junit4]   2> WARNING: Suite execution timed out:
>> org.apache.lucene.search.TestFuzzyQuery
>>    [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
>>    [junit4]   2>         at java.lang.Object.wait(Native Method)
>>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
>>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.run(RandomizedRunner.java:457)
>>    [junit4]   2>         at com.carrotsearch.ant.tasks.jun
>> it4.slave.SlaveMain.execute(SlaveMain.java:244)
>>    [junit4]   2>         at com.carrotsearch.ant.tasks.jun
>> it4.slave.SlaveMain.main(SlaveMain.java:355)
>>    [junit4]   2>         at com.carrotsearch.ant.tasks.jun
>> it4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
>>    [junit4]   2>    2) Thread[id=3728, name=SUITE-TestFuzzyQuery-seed#[2D2CD0D03EF67E59],
>> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>>    [junit4]   2>         at java.lang.Thread.getStackTrace
>> (Thread.java:1556)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
>>    [junit4]   2>         at java.security.AccessController.doPrivileged(Native
>> Method)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.access$200(RandomizedRunner.java:140)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$2.run(RandomizedRunner.java:598)
>>    [junit4]   2>    3) Thread[id=9, name=JUnit4-serializer-daemon,
>> state=TIMED_WAITING, group=main]
>>    [junit4]   2>         at java.lang.Thread.sleep(Native Method)
>>    [junit4]   2>         at com.carrotsearch.ant.tasks.jun
>> it4.events.Serializer$1.run(Serializer.java:50)
>>    [junit4]   2>    4) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
>> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedContext.current(RandomizedContext.java:134)
>>    [junit4]   2>         at org.apache.lucene.util.LuceneT
>> estCase.random(LuceneTestCase.java:748)
>>    [junit4]   2>         at org.apache.lucene.search.TestF
>> uzzyQuery.randomSimpleString(TestFuzzyQuery.java:503)
>>    [junit4]   2>         at org.apache.lucene.search.TestF
>> uzzyQuery.testRandom(TestFuzzyQuery.java:517)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccess
>> orImpl.invoke(NativeMethodAccessorImpl.java:62)
>>    [junit4]   2>         at sun.reflect.DelegatingMethodAc
>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   2>         at java.lang.reflect.Method.invok
>> e(Method.java:498)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>    [junit4]   2>         at org.apache.lucene.util.Abstrac
>> tBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMa
>> xFailures.java:64)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   2>         at org.apache.lucene.util.Abstrac
>> tBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>> ShadowingOrOverridesOnMethodsRule.java:40)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>> ShadowingOrOverridesOnMethodsRule.java:40)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMa
>> xFailures.java:64)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   2> NOTE: download the large Jenkins line-docs file by
>> running 'ant get-jenkins-line-docs' in the lucene directory.
>>    [junit4]   2> NOTE: reproduce with: ant test
>> -Dtestcase=TestFuzzyQuery -Dtests.method=testRandom
>> -Dtests.seed=2D2CD0D03EF67E59 -Dtests.multiplier=2 -Dtests.nightly=true
>> -Dtests.slow=true -Dtests.linedocsfile=/home/jen
>> kins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-
>> master/test-data/enwiki.random.lines.txt -Dtests.locale=de-GR
>> -Dtests.timezone=America/Hermosillo -Dtests.asserts=true
>> -Dtests.file.encoding=US-ASCII
>>    [junit4] ERROR   7200s J0 | TestFuzzyQuery.testRandom <<<
>>    [junit4]    > Throwable #1: java.lang.Exception: Test abandoned
>> because suite timeout was reached.
>>    [junit4]    >        at __randomizedtesting.SeedInfo.s
>> eed([2D2CD0D03EF67E59]:0)
>>    [junit4]   2> Feb 27, 2017 10:38:13 PM com.carrotsearch.randomizedtesting.ThreadLeakControl
>> checkThreadLeaks
>>    [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked
>> thread(s).
>>    [junit4]   2> Feb 27, 2017 10:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl
>> checkThreadLeaks
>>    [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at
>> org.apache.lucene.search.TestFuzzyQuery:
>>    [junit4]   2>    1) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
>> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>>    [junit4]   2>         at java.util.WeakHashMap.getTable
>> (WeakHashMap.java:350)
>>    [junit4]   2>         at java.util.WeakHashMap.getEntry
>> (WeakHashMap.java:427)
>>    [junit4]   2>         at java.util.WeakHashMap.contains
>> Key(WeakHashMap.java:417)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedContext.context(RandomizedContext.java:258)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedContext.current(RandomizedContext.java:134)
>>    [junit4]   2>         at org.apache.lucene.util.LuceneT
>> estCase.random(LuceneTestCase.java:748)
>>    [junit4]   2>         at org.apache.lucene.search.TestF
>> uzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
>>    [junit4]   2>         at org.apache.lucene.search.TestF
>> uzzyQuery.testRandom(TestFuzzyQuery.java:517)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccess
>> orImpl.invoke(NativeMethodAccessorImpl.java:62)
>>    [junit4]   2>         at sun.reflect.DelegatingMethodAc
>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   2>         at java.lang.reflect.Method.invok
>> e(Method.java:498)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>    [junit4]   2>         at org.apache.lucene.util.Abstrac
>> tBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMa
>> xFailures.java:64)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   2>         at org.apache.lucene.util.Abstrac
>> tBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>> ShadowingOrOverridesOnMethodsRule.java:40)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>> ShadowingOrOverridesOnMethodsRule.java:40)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMa
>> xFailures.java:64)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   2> Feb 27, 2017 10:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl
>> tryToInterruptAll
>>    [junit4]   2> INFO: Starting to interrupt leaked threads:
>>    [junit4]   2>    1) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
>> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>>    [junit4]   2> Feb 27, 2017 10:38:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl
>> tryToInterruptAll
>>    [junit4]   2> SEVERE: There are still zombie threads that couldn't be
>> terminated:
>>    [junit4]   2>    1) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
>> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>>    [junit4]   2>         at java.lang.Object.hashCode(Native Method)
>>    [junit4]   2>         at java.util.WeakHashMap.hash(Wea
>> kHashMap.java:298)
>>    [junit4]   2>         at java.util.WeakHashMap.get(Weak
>> HashMap.java:396)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedContext.getPerThread(RandomizedContext.java:319)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedContext.getRandomness(RandomizedContext.java:92)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedContext.getRandom(RandomizedContext.java:114)
>>    [junit4]   2>         at org.apache.lucene.util.LuceneT
>> estCase.random(LuceneTestCase.java:748)
>>    [junit4]   2>         at org.apache.lucene.search.TestF
>> uzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
>>    [junit4]   2>         at org.apache.lucene.search.TestF
>> uzzyQuery.testRandom(TestFuzzyQuery.java:517)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccess
>> orImpl.invoke(NativeMethodAccessorImpl.java:62)
>>    [junit4]   2>         at sun.reflect.DelegatingMethodAc
>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   2>         at java.lang.reflect.Method.invok
>> e(Method.java:498)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>>    [junit4]   2>         at org.apache.lucene.util.Abstrac
>> tBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMa
>> xFailures.java:64)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   2>         at org.apache.lucene.util.Abstrac
>> tBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>> ShadowingOrOverridesOnMethodsRule.java:40)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(No
>> ShadowingOrOverridesOnMethodsRule.java:40)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMa
>> xFailures.java:64)
>>    [junit4]   2>         at org.apache.lucene.util.TestRul
>> eIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtes
>> ting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   2> NOTE: test params are: codec=SimpleText,
>> sim=RandomSimilarity(queryNorm=true): {field=DFR I(n)B2}, locale=de-GR,
>> timezone=America/Hermosillo
>>    [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation
>> 1.8.0_121 (64-bit)/cpus=4,threads=2,free=103969688,total=250609664
>>    [junit4]   2> NOTE: All tests run in this JVM: [TestIndexInput,
>> TestDocsAndPositions, TestIndexSearcher, TestFlex,
>> TestAllFilesCheckIndexHeader, TestMaxTermFrequency,
>> TestDuelingCodecsAtNight, TestIndexWriterForceMerge, TestBinaryTerms,
>> TestMatchNoDocsQuery, TestIndexReaderClose, TestNoDeletionPolicy,
>> TestDoubleValuesSource, TestSimpleExplanationsWithFillerDocs,
>> TestTopFieldCollector, TestNamedSPILoader, TestMultiThreadTermVectors,
>> TestWordlistLoader, TestSparseFixedBitSet, TestDocValuesIndexing,
>> TestIndexWriterWithThreads, TestBagOfPositions, TestIndexWriterExceptions,
>> TestIndexWriterOnVMError, TestAutomatonQuery, TestBooleanMinShouldMatch,
>> TestRateLimiter, TestOrdinalMap, TestMSBRadixSorter, TestNot,
>> TestTermVectorsWriter, TestFuzzyQuery]
>>    [junit4]   2> NOTE: download the large Jenkins line-docs file by
>> running 'ant get-jenkins-line-docs' in the lucene directory.
>>    [junit4]   2> NOTE: reproduce with: ant test
>> -Dtestcase=TestFuzzyQuery -Dtests.seed=2D2CD0D03EF67E59
>> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
>> -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/
>> Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
>> -Dtests.locale=de-GR -Dtests.timezone=America/Hermosillo
>> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>>    [junit4] ERROR   0.00s J0 | TestFuzzyQuery (suite) <<<
>>    [junit4]    > Throwable #1: java.lang.Exception: Suite timeout
>> exceeded (>= 7200000 msec).
>>    [junit4]    >        at __randomizedtesting.SeedInfo.s
>> eed([2D2CD0D03EF67E59]:0)
>>    [junit4] Completed [405/448 (1!)] on J0 in 7223.14s, 3 tests, 2 errors
>> <<< FAILURES!
>>
>> [...truncated 74542 lines...]
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>

Re: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1250 - Still unstable

Posted by Adrien Grand <jp...@gmail.com>.
I pushed a fix. This was an infinite loop due to the fact that the test
tried to generate a number of unique terms that was greater than the
vocabulary size.

Le mar. 28 févr. 2017 à 11:43, Adrien Grand <jp...@gmail.com> a écrit :

> This reproduces for me, I'll dig.
>
> Le mar. 28 févr. 2017 à 09:58, Apache Jenkins Server <
> jenkins@builds.apache.org> a écrit :
>
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1250/
>
> 2 tests failed.
> FAILED:  org.apache.lucene.search.TestFuzzyQuery.testRandom
>
> Error Message:
> Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
>         at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>
>
> FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestFuzzyQuery
>
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>         at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>
>
>
>
> Build Log:
> [...truncated 1695 lines...]
>    [junit4] Suite: org.apache.lucene.search.TestFuzzyQuery
>    [junit4]   2> Feb 27, 2017 10:38:13 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
>    [junit4]   2> WARNING: Suite execution timed out:
> org.apache.lucene.search.TestFuzzyQuery
>    [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
>    [junit4]   2>         at java.lang.Object.wait(Native Method)
>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
>    [junit4]   2>    2) Thread[id=3728,
> name=SUITE-TestFuzzyQuery-seed#[2D2CD0D03EF67E59], state=RUNNABLE,
> group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at
> java.lang.Thread.getStackTrace(Thread.java:1556)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
>    [junit4]   2>         at
> java.security.AccessController.doPrivileged(Native Method)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
>    [junit4]   2>    3) Thread[id=9, name=JUnit4-serializer-daemon,
> state=TIMED_WAITING, group=main]
>    [junit4]   2>         at java.lang.Thread.sleep(Native Method)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
>    [junit4]   2>    4) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
>    [junit4]   2>         at
> org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:503)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at
> java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by
> running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery
> -Dtests.method=testRandom -Dtests.seed=2D2CD0D03EF67E59
> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
> -Dtests.locale=de-GR -Dtests.timezone=America/Hermosillo
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   7200s J0 | TestFuzzyQuery.testRandom <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because
> suite timeout was reached.
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>    [junit4]   2> Feb 27, 2017 10:38:13 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked
> thread(s).
>    [junit4]   2> Feb 27, 2017 10:38:33 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at
> org.apache.lucene.search.TestFuzzyQuery:
>    [junit4]   2>    1) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at
> java.util.WeakHashMap.getTable(WeakHashMap.java:350)
>    [junit4]   2>         at
> java.util.WeakHashMap.getEntry(WeakHashMap.java:427)
>    [junit4]   2>         at
> java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
>    [junit4]   2>         at
> org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at
> java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> Feb 27, 2017 10:38:33 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> INFO: Starting to interrupt leaked threads:
>    [junit4]   2>    1) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2> Feb 27, 2017 10:38:36 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> SEVERE: There are still zombie threads that couldn't be
> terminated:
>    [junit4]   2>    1) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at java.lang.Object.hashCode(Native Method)
>    [junit4]   2>         at
> java.util.WeakHashMap.hash(WeakHashMap.java:298)
>    [junit4]   2>         at java.util.WeakHashMap.get(WeakHashMap.java:396)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
>    [junit4]   2>         at
> org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at
> java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: test params are: codec=SimpleText,
> sim=RandomSimilarity(queryNorm=true): {field=DFR I(n)B2}, locale=de-GR,
> timezone=America/Hermosillo
>    [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation
> 1.8.0_121 (64-bit)/cpus=4,threads=2,free=103969688,total=250609664
>    [junit4]   2> NOTE: All tests run in this JVM: [TestIndexInput,
> TestDocsAndPositions, TestIndexSearcher, TestFlex,
> TestAllFilesCheckIndexHeader, TestMaxTermFrequency,
> TestDuelingCodecsAtNight, TestIndexWriterForceMerge, TestBinaryTerms,
> TestMatchNoDocsQuery, TestIndexReaderClose, TestNoDeletionPolicy,
> TestDoubleValuesSource, TestSimpleExplanationsWithFillerDocs,
> TestTopFieldCollector, TestNamedSPILoader, TestMultiThreadTermVectors,
> TestWordlistLoader, TestSparseFixedBitSet, TestDocValuesIndexing,
> TestIndexWriterWithThreads, TestBagOfPositions, TestIndexWriterExceptions,
> TestIndexWriterOnVMError, TestAutomatonQuery, TestBooleanMinShouldMatch,
> TestRateLimiter, TestOrdinalMap, TestMSBRadixSorter, TestNot,
> TestTermVectorsWriter, TestFuzzyQuery]
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by
> running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery
> -Dtests.seed=2D2CD0D03EF67E59 -Dtests.multiplier=2 -Dtests.nightly=true
> -Dtests.slow=true
> -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
> -Dtests.locale=de-GR -Dtests.timezone=America/Hermosillo
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   0.00s J0 | TestFuzzyQuery (suite) <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded
> (>= 7200000 msec).
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>    [junit4] Completed [405/448 (1!)] on J0 in 7223.14s, 3 tests, 2 errors
> <<< FAILURES!
>
> [...truncated 74542 lines...]
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 1250 - Still unstable

Posted by Adrien Grand <jp...@gmail.com>.
This reproduces for me, I'll dig.

Le mar. 28 févr. 2017 à 09:58, Apache Jenkins Server <
jenkins@builds.apache.org> a écrit :

> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1250/
>
> 2 tests failed.
> FAILED:  org.apache.lucene.search.TestFuzzyQuery.testRandom
>
> Error Message:
> Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
>         at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>
>
> FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestFuzzyQuery
>
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>         at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>
>
>
>
> Build Log:
> [...truncated 1695 lines...]
>    [junit4] Suite: org.apache.lucene.search.TestFuzzyQuery
>    [junit4]   2> Feb 27, 2017 10:38:13 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
>    [junit4]   2> WARNING: Suite execution timed out:
> org.apache.lucene.search.TestFuzzyQuery
>    [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
>    [junit4]   2>         at java.lang.Object.wait(Native Method)
>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
>    [junit4]   2>    2) Thread[id=3728,
> name=SUITE-TestFuzzyQuery-seed#[2D2CD0D03EF67E59], state=RUNNABLE,
> group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at
> java.lang.Thread.getStackTrace(Thread.java:1556)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
>    [junit4]   2>         at
> java.security.AccessController.doPrivileged(Native Method)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
>    [junit4]   2>    3) Thread[id=9, name=JUnit4-serializer-daemon,
> state=TIMED_WAITING, group=main]
>    [junit4]   2>         at java.lang.Thread.sleep(Native Method)
>    [junit4]   2>         at
> com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
>    [junit4]   2>    4) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
>    [junit4]   2>         at
> org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:503)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at
> java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by
> running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery
> -Dtests.method=testRandom -Dtests.seed=2D2CD0D03EF67E59
> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
> -Dtests.locale=de-GR -Dtests.timezone=America/Hermosillo
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   7200s J0 | TestFuzzyQuery.testRandom <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because
> suite timeout was reached.
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>    [junit4]   2> Feb 27, 2017 10:38:13 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked
> thread(s).
>    [junit4]   2> Feb 27, 2017 10:38:33 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at
> org.apache.lucene.search.TestFuzzyQuery:
>    [junit4]   2>    1) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at
> java.util.WeakHashMap.getTable(WeakHashMap.java:350)
>    [junit4]   2>         at
> java.util.WeakHashMap.getEntry(WeakHashMap.java:427)
>    [junit4]   2>         at
> java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
>    [junit4]   2>         at
> org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at
> java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> Feb 27, 2017 10:38:33 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> INFO: Starting to interrupt leaked threads:
>    [junit4]   2>    1) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2> Feb 27, 2017 10:38:36 PM
> com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> SEVERE: There are still zombie threads that couldn't be
> terminated:
>    [junit4]   2>    1) Thread[id=3729,
> name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59],
> state=RUNNABLE, group=TGRP-TestFuzzyQuery]
>    [junit4]   2>         at java.lang.Object.hashCode(Native Method)
>    [junit4]   2>         at
> java.util.WeakHashMap.hash(WeakHashMap.java:298)
>    [junit4]   2>         at java.util.WeakHashMap.get(WeakHashMap.java:396)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
>    [junit4]   2>         at
> org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
>    [junit4]   2>         at
> org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at
> java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: test params are: codec=SimpleText,
> sim=RandomSimilarity(queryNorm=true): {field=DFR I(n)B2}, locale=de-GR,
> timezone=America/Hermosillo
>    [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation
> 1.8.0_121 (64-bit)/cpus=4,threads=2,free=103969688,total=250609664
>    [junit4]   2> NOTE: All tests run in this JVM: [TestIndexInput,
> TestDocsAndPositions, TestIndexSearcher, TestFlex,
> TestAllFilesCheckIndexHeader, TestMaxTermFrequency,
> TestDuelingCodecsAtNight, TestIndexWriterForceMerge, TestBinaryTerms,
> TestMatchNoDocsQuery, TestIndexReaderClose, TestNoDeletionPolicy,
> TestDoubleValuesSource, TestSimpleExplanationsWithFillerDocs,
> TestTopFieldCollector, TestNamedSPILoader, TestMultiThreadTermVectors,
> TestWordlistLoader, TestSparseFixedBitSet, TestDocValuesIndexing,
> TestIndexWriterWithThreads, TestBagOfPositions, TestIndexWriterExceptions,
> TestIndexWriterOnVMError, TestAutomatonQuery, TestBooleanMinShouldMatch,
> TestRateLimiter, TestOrdinalMap, TestMSBRadixSorter, TestNot,
> TestTermVectorsWriter, TestFuzzyQuery]
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by
> running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery
> -Dtests.seed=2D2CD0D03EF67E59 -Dtests.multiplier=2 -Dtests.nightly=true
> -Dtests.slow=true
> -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
> -Dtests.locale=de-GR -Dtests.timezone=America/Hermosillo
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   0.00s J0 | TestFuzzyQuery (suite) <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded
> (>= 7200000 msec).
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
>    [junit4] Completed [405/448 (1!)] on J0 in 7223.14s, 3 tests, 2 errors
> <<< FAILURES!
>
> [...truncated 74542 lines...]
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1250 - Still unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1250/

2 tests failed.
FAILED:  org.apache.lucene.search.TestFuzzyQuery.testRandom

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestFuzzyQuery

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)




Build Log:
[...truncated 1695 lines...]
   [junit4] Suite: org.apache.lucene.search.TestFuzzyQuery
   [junit4]   2> Feb 27, 2017 10:38:13 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.search.TestFuzzyQuery
   [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    2) Thread[id=3728, name=SUITE-TestFuzzyQuery-seed#[2D2CD0D03EF67E59], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1556)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
   [junit4]   2>    3) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2>    4) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
   [junit4]   2>         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:503)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery -Dtests.method=testRandom -Dtests.seed=2D2CD0D03EF67E59 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=de-GR -Dtests.timezone=America/Hermosillo -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   7200s J0 | TestFuzzyQuery.testRandom <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
   [junit4]   2> Feb 27, 2017 10:38:13 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 27, 2017 10:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.search.TestFuzzyQuery: 
   [junit4]   2>    1) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at java.util.WeakHashMap.getTable(WeakHashMap.java:350)
   [junit4]   2>         at java.util.WeakHashMap.getEntry(WeakHashMap.java:427)
   [junit4]   2>         at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
   [junit4]   2>         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Feb 27, 2017 10:38:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2> Feb 27, 2017 10:38:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=3729, name=TEST-TestFuzzyQuery.testRandom-seed#[2D2CD0D03EF67E59], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at java.lang.Object.hashCode(Native Method)
   [junit4]   2>         at java.util.WeakHashMap.hash(WeakHashMap.java:298)
   [junit4]   2>         at java.util.WeakHashMap.get(WeakHashMap.java:396)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
   [junit4]   2>         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarity(queryNorm=true): {field=DFR I(n)B2}, locale=de-GR, timezone=America/Hermosillo
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=103969688,total=250609664
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexInput, TestDocsAndPositions, TestIndexSearcher, TestFlex, TestAllFilesCheckIndexHeader, TestMaxTermFrequency, TestDuelingCodecsAtNight, TestIndexWriterForceMerge, TestBinaryTerms, TestMatchNoDocsQuery, TestIndexReaderClose, TestNoDeletionPolicy, TestDoubleValuesSource, TestSimpleExplanationsWithFillerDocs, TestTopFieldCollector, TestNamedSPILoader, TestMultiThreadTermVectors, TestWordlistLoader, TestSparseFixedBitSet, TestDocValuesIndexing, TestIndexWriterWithThreads, TestBagOfPositions, TestIndexWriterExceptions, TestIndexWriterOnVMError, TestAutomatonQuery, TestBooleanMinShouldMatch, TestRateLimiter, TestOrdinalMap, TestMSBRadixSorter, TestNot, TestTermVectorsWriter, TestFuzzyQuery]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery -Dtests.seed=2D2CD0D03EF67E59 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=de-GR -Dtests.timezone=America/Hermosillo -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | TestFuzzyQuery (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2D2CD0D03EF67E59]:0)
   [junit4] Completed [405/448 (1!)] on J0 in 7223.14s, 3 tests, 2 errors <<< FAILURES!

[...truncated 74542 lines...]



[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1249 - Failure

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1249/

3 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Error from server at http://127.0.0.1:53607/h_nz/as/collection1: this IndexWriter is closed

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:53607/h_nz/as/collection1: this IndexWriter is closed
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
	at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:1538)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:664)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 44 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper  at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)  at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)  at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)  at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)  at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)  at java.lang.Thread.run(Thread.java:745)  

Stack Trace:
java.lang.AssertionError: ObjectTracker found 44 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper]
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)

org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.RawDirectoryWrapper
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
	at org.apache.solr.core.MetricsDirectoryFactory.get(MetricsDirectoryFactory.java:208)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:388)
	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:269)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:399)
	at org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:281)
	at java.lang.Thread.run(Thread.java:745)


	at __randomizedtesting.SeedInfo.seed([FF46665015CA08DE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:302)
	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@636c0f3d (not leader) wrong [docid] for SolrDocument{id=0, id_field_copy_that_does_not_support_in_place_update_s=0, title_s=title0, id_i=0, inplace_updatable_float=101.0, _version_=1560484088038555648, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=45} expected:<12540> but was:<45>

Stack Trace:
java.lang.AssertionError: 'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@636c0f3d (not leader) wrong [docid] for SolrDocument{id=0, id_field_copy_that_does_not_support_in_place_update_s=0, title_s=title0, id_i=0, inplace_updatable_float=101.0, _version_=1560484088038555648, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=45} expected:<12540> but was:<45>
	at __randomizedtesting.SeedInfo.seed([FF46665015CA08DE:7712598ABB366526]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesInResults(TestInPlaceUpdatesDistrib.java:442)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesAgainstAllClients(TestInPlaceUpdatesDistrib.java:413)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:321)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12205 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_FF46665015CA08DE-001/init-core-data-001
   [junit4]   2> 801727 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[FF46665015CA08DE]-worker) [    ] o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 801728 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[FF46665015CA08DE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 801728 INFO  (SUITE-FullSolrCloudDistribCmdsTest-seed#[FF46665015CA08DE]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /h_nz/as
   [junit4]   2> 801732 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 801733 INFO  (Thread-632) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 801733 INFO  (Thread-632) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 801833 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.ZkTestServer start zk server on port:39659
   [junit4]   2> 801856 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 801858 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 801859 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 801860 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 801860 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 801861 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 801862 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 801863 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 801864 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 801864 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 801865 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 801942 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_FF46665015CA08DE-001/control-001/cores/collection1
   [junit4]   2> 801943 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 801945 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15230a0c{/h_nz/as,null,AVAILABLE}
   [junit4]   2> 801946 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@689492d2{HTTP/1.1,[http/1.1]}{127.0.0.1:53607}
   [junit4]   2> 801946 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.e.j.s.Server Started @866164ms
   [junit4]   2> 801946 INFO  (TEST-FullSolrCloudDistribCmdsTest.test-seed#[FF46665015CA08DE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest_FF46665015CA08DE-001/tempDir-001/control/data, hostContext=/h_nz/as, hostPort=53607, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistri

[...truncated too long message...]

linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=pt-PT -Dtests.timezone=America/Godthab -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE  105s J2 | TestInPlaceUpdatesDistrib.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: 'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@636c0f3d (not leader) wrong [docid] for SolrDocument{id=0, id_field_copy_that_does_not_support_in_place_update_s=0, title_s=title0, id_i=0, inplace_updatable_float=101.0, _version_=1560484088038555648, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=45} expected:<12540> but was:<45>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FF46665015CA08DE:7712598ABB366526]:0)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesInResults(TestInPlaceUpdatesDistrib.java:442)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesAgainstAllClients(TestInPlaceUpdatesDistrib.java:413)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:321)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:140)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4589048 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[FF46665015CA08DE]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 4589048 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[FF46665015CA08DE]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1598198612
   [junit4]   2> 4589048 INFO  (coreCloseExecutor-13810-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3e380db4
   [junit4]   2> 4589107 INFO  (coreCloseExecutor-13810-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 4589109 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[FF46665015CA08DE]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.update.TestInPlaceUpdatesDistrib_FF46665015CA08DE-001
   [junit4]   2> Feb 27, 2017 11:07:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=9, maxDocsPerChunk=881, blockSize=258), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=9, blockSize=258)), sim=RandomSimilarity(queryNorm=false): {}, locale=pt-PT, timezone=America/Godthab
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=170599288,total=522715136
   [junit4]   2> NOTE: All tests run in this JVM: [TestNumericTerms64, TestReload, TestInPlaceUpdatesStandalone, SolrCoreCheckLockOnStartupTest, TestUtils, SolrGraphiteReporterTest, TestSolrQueryParserDefaultOperatorResource, OverseerRolesTest, TestFieldTypeResource, SolrIndexConfigTest, SuggesterTSTTest, ReplaceNodeTest, DeleteStatusTest, TestComponentsName, TestSolrQueryParser, TestCloudRecovery, TestGroupingSearch, TestManagedResource, AnalyticsMergeStrategyTest, PreAnalyzedFieldManagedSchemaCloudTest, HLLSerializationTest, RequestLoggingTest, SolrPluginUtilsTest, TestStressLiveNodes, TestAddFieldRealTimeGet, BasicAuthIntegrationTest, RequestHandlersTest, SuggesterWFSTTest, TestFieldCollectionResource, TestInitParams, ShowFileRequestHandlerTest, TestExtendedDismaxParser, SolrTestCaseJ4Test, TestBulkSchemaAPI, TestSha256AuthenticationProvider, DocValuesNotIndexedTest, TestRandomFlRTGCloud, TestCollationFieldDocValues, TestShardHandlerFactory, HardAutoCommitTest, FullSolrCloudDistribCmdsTest, DistribJoinFromCollectionTest, StatsComponentTest, TestConfig, HdfsBasicDistributedZk2Test, TestDocSet, PreAnalyzedUpdateProcessorTest, TestLocalFSCloudBackupRestore, TestComplexPhraseQParserPlugin, TestSizeLimitedDistributedMap, TestInitQParser, CustomCollectionTest, MBeansHandlerTest, TestStandardQParsers, ResourceLoaderTest, SecurityConfHandlerTest, FieldAnalysisRequestHandlerTest, TestExceedMaxTermLength, TestStressReorder, TestLuceneMatchVersion, TestTolerantSearch, CollectionReloadTest, TestConfigOverlay, XmlUpdateRequestHandlerTest, TestAuthorizationFramework, TestPHPSerializedResponseWriter, TestExactSharedStatsCache, TestApiFramework, TestMacroExpander, TestCoreDiscovery, TestLeaderElectionWithEmptyReplica, TestCryptoKeys, DistributedFacetExistsSmallTest, TestManagedSchemaThreadSafety, AnalysisErrorHandlingTest, TermVectorComponentDistributedTest, ExternalFileFieldSortTest, DirectUpdateHandlerTest, TestHdfsBackupRestoreCore, HdfsUnloadDistributedZkTest, SubstringBytesRefFilterTest, SuggestComponentTest, TestFilteredDocIdSet, TestFieldCache, CollectionsAPIAsyncDistributedZkTest, TestPivotHelperCode, TestSolrCloudWithHadoopAuthPlugin, LeaderElectionTest, ConfigSetsAPITest, TestRestoreCore, TestSolrCloudWithSecureImpersonation, DistributedQueryComponentCustomSortTest, TestSolrConfigHandlerCloud, HdfsChaosMonkeySafeLeaderTest, TestJsonFacets, TestCloudManagedSchema, SimpleCollectionCreateDeleteTest, DebugComponentTest, TestRecovery, NoCacheHeaderTest, CoreMergeIndexesAdminHandlerTest, RecoveryZkTest, MissingSegmentRecoveryTest, TestIBSimilarityFactory, TestPhraseSuggestions, TestJettySolrRunner, SliceStateTest, TestSort, SimpleFacetsTest, TestCollationField, TestLMDirichletSimilarityFactory, HdfsDirectoryTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SolrXmlInZkTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudPivotFacet, TestConfigSetsAPI, TestDownShardTolerantSearch, TestHdfsCloudBackupRestore, TestLeaderElectionZkExpiry, TestLockTree, TestRebalanceLeaders, TestReplicaProperties, TestRequestStatusCollectionAPI, TestSSLRandomization, TestSegmentSorting, TestShortCircuitedRequests, TestSolrCloudWithDelegationTokens, TestStressCloudBlindAtomicUpdates, HdfsRecoverLeaseTest, HdfsRestartWhileUpdatingTest, HdfsWriteToMultipleCollectionsTest, TestClusterStateMutator, ZkStateReaderTest, CoreSorterTest, ExitableDirectoryReaderTest, HdfsDirectoryFactoryTest, TestInfoStreamLogging, TestNRTOpen, TestReloadAndDeleteDocs, TestSolrConfigHandler, TestSolrDynamicMBean, TestSolrXml, TestBlobHandler, TestReqParamsAPI, TestSQLHandler, DistributedFacetPivotSmallAdvancedTest, DistributedFacetPivotSmallTest, FacetPivotSmallTest, ShufflingReplicaListTransformerTest, SpatialHeatmapFacetsTest, TestIntervalFaceting, TestManagedResourceStorage, TestDefaultSearchFieldResource, TestDynamicFieldCollectionResource, TestDynamicFieldResource, TestFieldResource, TestSchemaNameResource, TestSchemaResource, TestSchemaSimilarityResource, TestSchemaVersionResource, TestSerializedLuceneMatchVersion, TestSolrQueryParserResource, TestUniqueKeyFieldResource, TestManagedStopFilterFactory, BadCopyFieldTest, BooleanFieldTest, ChangedSchemaMergeTest, DateRangeFieldTest, TestBulkSchemaConcurrent, TestManagedSchemaAPI, TestSchemalessBufferedUpdates, TestUseDocValuesAsStored, MergeStrategyTest, SortSpecParsingTest, TestComplexPhraseLeadingWildcard, TestCustomSort, TestGraphTermsQParserPlugin, TestMissingGroups, TestReRankQParserPlugin, TestReloadDeadlock, TestSearcherReuse, TestSolr4Spatial2, TestSolrFieldCacheMBean, TestStressUserVersions, TestXmlQParser, TestXmlQParserPlugin, TestOrdValues, TestSortByMinMaxFunction, BlockJoinFacetDistribTest, BlockJoinFacetRandomTest, BlockJoinFacetSimpleTest, GraphQueryTest, TestScoreJoinQPNoScore, TestJsonRequest, CloudMLTQParserTest, TestClassicSimilarityFactory, TestDFISimilarityFactory, TestExactStatsCache, BufferStoreTest, HdfsLockFactoryTest, TestFieldCacheWithThreads, TestLegacyFieldCache, TestNumericTerms32, PeerSyncWithIndexFingerprintCachingTest, SolrIndexMetricsTest, TestDocBasedVersionConstraints, TestInPlaceUpdatesDistrib]
   [junit4] Completed [693/693 (3!)] on J2 in 105.71s, 1 test, 1 failure <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J2-20170227_095023_2006829828118071801380.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid17550.hprof ...
   [junit4] Heap dump file created [572401605 bytes in 34.890 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 11692 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:783: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:727: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid17550.hprof

Total time: 273 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1248 - Still unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1248/

2 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.test

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
	at __randomizedtesting.SeedInfo.seed([1D48FA3EBA96D4D4:951CC5E4146AB92C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1251)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:702)
	at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:122)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig

Error Message:
The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.

Stack Trace:
java.lang.RuntimeException: The max direct memory is likely too low.  Either increase it (by adding -XX:MaxDirectMemorySize=<size>g -XX:+UseLargePages to your containers startup args) or disable direct allocation using solr.hdfs.blockcache.direct.memory.allocation=false in solrconfig.xml. If you are putting the block cache on the heap, your java heap size might not be large enough. Failed allocating ~134.217728 MB.
	at __randomizedtesting.SeedInfo.seed([1D48FA3EBA96D4D4:EAE73315471F3EFF]:0)
	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:310)
	at org.apache.solr.core.HdfsDirectoryFactory.getBlockDirectoryCache(HdfsDirectoryFactory.java:286)
	at org.apache.solr.core.HdfsDirectoryFactory.create(HdfsDirectoryFactory.java:226)
	at org.apache.solr.core.HdfsDirectoryFactoryTest.testInitArgsOrSysPropConfig(HdfsDirectoryFactoryTest.java:108)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:693)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at org.apache.solr.store.blockcache.BlockCache.<init>(BlockCache.java:70)
	at org.apache.solr.core.HdfsDirectoryFactory.createBlockCache(HdfsDirectoryFactory.java:308)
	... 42 more




Build Log:
[...truncated 11773 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_1D48FA3EBA96D4D4-001/init-core-data-001
   [junit4]   2> 495875 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 495881 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 495882 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_1D48FA3EBA96D4D4-001/tempDir-001
   [junit4]   2> 495882 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 495897 INFO  (Thread-23061) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 495897 INFO  (Thread-23061) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 495997 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:49279
   [junit4]   2> 496029 INFO  (jetty-launcher-1204-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 496033 INFO  (jetty-launcher-1204-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 496034 INFO  (jetty-launcher-1204-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@400580f7{/solr,null,AVAILABLE}
   [junit4]   2> 496035 INFO  (jetty-launcher-1204-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@765774c9{HTTP/1.1,[http/1.1]}{127.0.0.1:50578}
   [junit4]   2> 496035 INFO  (jetty-launcher-1204-thread-1) [    ] o.e.j.s.Server Started @498738ms
   [junit4]   2> 496035 INFO  (jetty-launcher-1204-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50578}
   [junit4]   2> 496035 ERROR (jetty-launcher-1204-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 496035 INFO  (jetty-launcher-1204-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 496035 INFO  (jetty-launcher-1204-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 496035 INFO  (jetty-launcher-1204-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 496035 INFO  (jetty-launcher-1204-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-26T08:12:04.214Z
   [junit4]   2> 496036 INFO  (jetty-launcher-1204-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@39f1c243{/solr,null,AVAILABLE}
   [junit4]   2> 496037 INFO  (jetty-launcher-1204-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@3201ef46{HTTP/1.1,[http/1.1]}{127.0.0.1:46278}
   [junit4]   2> 496037 INFO  (jetty-launcher-1204-thread-2) [    ] o.e.j.s.Server Started @498740ms
   [junit4]   2> 496037 INFO  (jetty-launcher-1204-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46278}
   [junit4]   2> 496037 ERROR (jetty-launcher-1204-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 496037 INFO  (jetty-launcher-1204-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 496037 INFO  (jetty-launcher-1204-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 496037 INFO  (jetty-launcher-1204-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 496037 INFO  (jetty-launcher-1204-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-26T08:12:04.216Z
   [junit4]   2> 496047 INFO  (jetty-launcher-1204-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 496059 INFO  (jetty-launcher-1204-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 496077 INFO  (jetty-launcher-1204-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49279/solr
   [junit4]   2> 496109 INFO  (jetty-launcher-1204-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49279/solr
   [junit4]   2> 496222 INFO  (jetty-launcher-1204-thread-1) [n:127.0.0.1:50578_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 496231 INFO  (jetty-launcher-1204-thread-2) [n:127.0.0.1:46278_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 496231 INFO  (jetty-launcher-1204-thread-1) [n:127.0.0.1:50578_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50578_solr
   [junit4]   2> 496232 INFO  (jetty-launcher-1204-thread-2) [n:127.0.0.1:46278_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46278_solr
   [junit4]   2> 496232 INFO  (jetty-launcher-1204-thread-1) [n:127.0.0.1:50578_solr    ] o.a.s.c.Overseer Overseer (id=97523906909831174-127.0.0.1:50578_solr-n_0000000000) starting
   [junit4]   2> 496296 INFO  (OverseerStateUpdate-97523906909831174-127.0.0.1:50578_solr-n_0000000000) [n:127.0.0.1:50578_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 496300 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:46278_solr) [n:127.0.0.1:46278_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 496306 INFO  (jetty-launcher-1204-thread-1) [n:127.0.0.1:50578_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50578_solr
   [junit4]   2> 496307 INFO  (zkCallback-1216-thread-1-processing-n:127.0.0.1:50578_solr) [n:127.0.0.1:50578_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 496307 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:46278_solr) [n:127.0.0.1:46278_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 496403 INFO  (jetty-launcher-1204-thread-2) [n:127.0.0.1:46278_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_1D48FA3EBA96D4D4-001/tempDir-001/node2/.
   [junit4]   2> 496406 INFO  (jetty-launcher-1204-thread-1) [n:127.0.0.1:50578_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_1D48FA3EBA96D4D4-001/tempDir-001/node1/.
   [junit4]   2> 496479 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 496480 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49279/solr ready
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 496510 WARN  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 496514 WARN  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 496516 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 496526 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_59795_hdfs____rby5g9/webapp
   [junit4]   2> 497114 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59795
   [junit4]   2> 497619 WARN  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 497620 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 497643 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_53622_datanode____.8pd85d/webapp
   [junit4]   2> 498125 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53622
   [junit4]   2> 498231 WARN  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 498232 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 498266 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_49039_datanode____aggymw/webapp
   [junit4]   2> 498382 INFO  (IPC Server handler 4 on 49187) [    ] BlockStateChange BLOCK* processReport: from storage DS-b058da5c-2148-4cab-af31-4e67b893f9f4 node DatanodeRegistration(127.0.0.1:52679, datanodeUuid=084e0297-742f-4300-8dc5-ef3c5f07ade7, infoPort=34082, infoSecurePort=0, ipcPort=53956, storageInfo=lv=-56;cid=testClusterID;nsid=1452967859;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs
   [junit4]   2> 498382 INFO  (IPC Server handler 4 on 49187) [    ] BlockStateChange BLOCK* processReport: from storage DS-7255cba6-f405-4604-8cd7-341382a7f66e node DatanodeRegistration(127.0.0.1:52679, datanodeUuid=084e0297-742f-4300-8dc5-ef3c5f07ade7, infoPort=34082, infoSecurePort=0, ipcPort=53956, storageInfo=lv=-56;cid=testClusterID;nsid=1452967859;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 498780 INFO  (SUITE-HdfsRecoveryZkTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49039
   [junit4]   2> 499110 INFO  (IPC Server handler 7 on 49187) [    ] BlockStateChange BLOCK* processReport: from storage DS-5554f715-28c8-48db-b64f-b877c95d6bb5 node DatanodeRegistration(127.0.0.1:59077, datanodeUuid=ee48a3fa-209c-4ab0-a921-4c8a96a81347, infoPort=50990, infoSecurePort=0, ipcPort=44118, storageInfo=lv=-56;cid=testClusterID;nsid=1452967859;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 499110 INFO  (IPC Server handler 7 on 49187) [    ] BlockStateChange BLOCK* processReport: from storage DS-4381a2dc-e1e3-4142-a656-31294bf444a6 node DatanodeRegistration(127.0.0.1:59077, datanodeUuid=ee48a3fa-209c-4ab0-a921-4c8a96a81347, infoPort=50990, infoSecurePort=0, ipcPort=44118, storageInfo=lv=-56;cid=testClusterID;nsid=1452967859;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 499205 INFO  (TEST-HdfsRecoveryZkTest.test-seed#[1D48FA3EBA96D4D4]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 499208 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 499233 INFO  (OverseerThreadFactory-1007-thread-1-processing-n:127.0.0.1:50578_solr) [n:127.0.0.1:50578_solr    ] o.a.s.c.CreateCollectionCmd Create collection recoverytest
   [junit4]   2> 499378 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 499379 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 499519 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:46278_solr) [n:127.0.0.1:46278_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 499520 INFO  (zkCallback-1216-thread-4-processing-n:127.0.0.1:50578_solr) [n:127.0.0.1:50578_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 500409 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 500412 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 500424 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema [recoverytest_shard1_replica2] Schema name=minimal
   [junit4]   2> 500427 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 500428 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'recoverytest_shard1_replica2' using configuration from collection recoverytest
   [junit4]   2> 500436 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema [recoverytest_shard1_replica1] Schema name=minimal
   [junit4]   2> 500443 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49187/data
   [junit4]   2> 500443 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 500443 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore [[recoverytest_shard1_replica2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_1D48FA3EBA96D4D4-001/tempDir-001/node1/recoverytest_shard1_replica2], dataDir=[hdfs://localhost:49187/data/recoverytest/core_node2/data/]
   [junit4]   2> 500443 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@259399c5
   [junit4]   2> 500443 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49187/data/recoverytest/core_node2/data/snapshot_metadata
   [junit4]   2> 500444 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 500447 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'recoverytest_shard1_replica1' using configuration from collection recoverytest
   [junit4]   2> 500447 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49187/data
   [junit4]   2> 500447 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 500447 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore [[recoverytest_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_1D48FA3EBA96D4D4-001/tempDir-001/node2/recoverytest_shard1_replica1], dataDir=[hdfs://localhost:49187/data/recoverytest/core_node1/data/]
   [junit4]   2> 500462 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@259399c5
   [junit4]   2> 500462 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49187/data/recoverytest/core_node1/data/snapshot_metadata
   [junit4]   2> 500476 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 500477 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 500477 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 500481 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 500481 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 500481 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 500495 WARN  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 500497 WARN  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 505507 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 505516 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 505516 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49187/data/recoverytest/core_node1/data
   [junit4]   2> 505518 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49187/data/recoverytest/core_node2/data
   [junit4]   2> 505566 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49187/data/recoverytest/core_node2/data/index
   [junit4]   2> 505572 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49187/data/recoverytest/core_node1/data/index
   [junit4]   2> 505584 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 505584 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 505584 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 505587 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 505587 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 505587 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 505610 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 505616 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 505684 INFO  (IPC Server handler 7 on 49187) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59077 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b058da5c-2148-4cab-af31-4e67b893f9f4:NORMAL:127.0.0.1:52679|RBW], ReplicaUC[[DISK]DS-5554f715-28c8-48db-b64f-b877c95d6bb5:NORMAL:127.0.0.1:59077|RBW]]} size 0
   [junit4]   2> 505684 INFO  (IPC Server handler 8 on 49187) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52679 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b058da5c-2148-4cab-af31-4e67b893f9f4:NORMAL:127.0.0.1:52679|RBW], ReplicaUC[[DISK]DS-5554f715-28c8-48db-b64f-b877c95d6bb5:NORMAL:127.0.0.1:59077|RBW]]} size 0
   [junit4]   2> 505742 INFO  (IPC Server handler 7 on 49187) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59077 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b058da5c-2148-4cab-af31-4e67b893f9f4:NORMAL:127.0.0.1:52679|RBW], ReplicaUC[[DISK]DS-4381a2dc-e1e3-4142-a656-31294bf444a6:NORMAL:127.0.0.1:59077|RBW]]} size 0
   [junit4]   2> 505743 INFO  (IPC Server handler 8 on 49187) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52679 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4381a2dc-e1e3-4142-a656-31294bf444a6:NORMAL:127.0.0.1:59077|RBW], ReplicaUC[[DISK]DS-7255cba6-f405-4604-8cd7-341382a7f66e:NORMAL:127.0.0.1:52679|FINALIZED]]} size 0
   [junit4]   2> 505833 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 505833 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 505833 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 505921 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 505921 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 505921 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 505928 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 505928 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 505994 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@8d52f19[recoverytest_shard1_replica1] main]
   [junit4]   2> 505995 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 505995 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 505995 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 506000 INFO  (searcherExecutor-1013-thread-1-processing-n:127.0.0.1:46278_solr x:recoverytest_shard1_replica1 s:shard1 c:recoverytest) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore [recoverytest_shard1_replica1] Registered new searcher Searcher@8d52f19[recoverytest_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 506006 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560382521144770560
   [junit4]   2> 506011 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 506011 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 506012 INFO  (zkCallback-1216-thread-2-processing-n:127.0.0.1:50578_solr) [n:127.0.0.1:50578_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 506013 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:46278_solr) [n:127.0.0.1:46278_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 506025 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 506041 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@7928bcdd[recoverytest_shard1_replica2] main]
   [junit4]   2> 506043 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 506044 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 506044 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 506046 INFO  (searcherExecutor-1012-thread-1-processing-n:127.0.0.1:50578_solr x:recoverytest_shard1_replica2 s:shard1 c:recoverytest) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore [recoverytest_shard1_replica2] Registered new searcher Searcher@7928bcdd[recoverytest_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 506048 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560382521188810752
   [junit4]   2> 506525 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 506525 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 506525 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/
   [junit4]   2> 506525 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica1 url=http://127.0.0.1:46278/solr START replicas=[http://127.0.0.1:50578/solr/recoverytest_shard1_replica2/] nUpdates=100
   [junit4]   2> 506532 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica1 url=http://127.0.0.1:46278/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 506534 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.c.S.Request [recoverytest_shard1_replica2]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 506544 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 506544 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 506544 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 506547 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/ shard1
   [junit4]   2> 506648 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:46278_solr) [n:127.0.0.1:46278_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 506649 INFO  (zkCallback-1216-thread-2-processing-n:127.0.0.1:50578_solr) [n:127.0.0.1:50578_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 506698 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 506701 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2} status=0 QTime=7323
   [junit4]   2> 506802 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:46278_solr) [n:127.0.0.1:46278_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 506803 INFO  (zkCallback-1216-thread-3-processing-n:127.0.0.1:50578_solr) [n:127.0.0.1:50578_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 507056 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2} status=0 QTime=7676
   [junit4]   2> 507061 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 507161 INFO  (zkCallback-1215-thread-1-processing-n:127.0.0.1:46278_solr) [n:127.0.0.1:46278_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 507162 INFO  (zkCallback-1216-thread-3-processing-n:127.0.0.1:50578_solr) [n:127.0.0.1:50578_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/recoverytest/state.json] for collection [recoverytest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 507238 INFO  (OverseerCollectionConfigSetProcessor-97523906909831174-127.0.0.1:50578_solr-n_0000000000) [n:127.0.0.1:50578_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 508061 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=8853
   [junit4]   2> 508081 INFO  (TEST-HdfsRecoveryZkTest.test-seed#[1D48FA3EBA96D4D4]) [    ] o.a.s.c.RecoveryZkTest Indexing 30000 documents
   [junit4]   2> 508145 DEBUG (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.HdfsTransactionLog Opening new tlog hdfs tlog{file=hdfs://localhost:49187/data/recoverytest/core_node1/data/tlog/tlog.0000000000000000000 refcount=1}
   [junit4]   2> 508177 DEBUG (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.HdfsTransactionLog Opening new tlog hdfs tlog{file=hdfs://localhost:49187/data/recoverytest/core_node2/data/tlog/tlog.0000000000000000000 refcount=1}
   [junit4]   2> 508200 INFO  (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-0 (1560382523375091712)]} 0 42
   [junit4]   2> 508201 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-0 (1560382523375091712)]} 0 68
   [junit4]   2> 508202 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-0 (1560382523377188864)]} 0 13
   [junit4]   2> 508202 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-0 (1560382523377188864)]} 0 67
   [junit4]   2> 508203 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-1 (1560382523447443456)]} 0 0
   [junit4]   2> 508203 INFO  (qtp2059649094-26212) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-1 (1560382523447443456)]} 0 1
   [junit4]   2> 508205 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-1 (1560382523448492032)]} 0 1
   [junit4]   2> 508205 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-0 (-1560382523449540608)]} 0 0
   [junit4]   2> 508205 INFO  (qtp2059649094-26204) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-1 (1560382523448492032)]} 0 2
   [junit4]   2> 508206 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-0 (-1560382523449540608)]} 0 2
   [junit4]   2> 508206 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-0 (-1560382523451637760)]} 0 0
   [junit4]   2> 508207 INFO  (qtp2059649094-26211) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-0 (-1560382523451637760)]} 0 1
   [junit4]   2> 508212 INFO  (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-2 (1560382523452686336)]} 0 0
   [junit4]   2> 508212 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-2 (1560382523452686336)]} 0 5
   [junit4]   2> 508213 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-2 (1560382523452686337)]} 0 1
   [junit4]   2> 508213 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-3 (1560382523457929216)]} 0 0
   [junit4]   2> 508213 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-2 (1560382523452686337)]} 0 6
   [junit4]   2> 508221 INFO  (qtp2059649094-26212) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-3 (1560382523457929216)]} 0 8
   [junit4]   2> 508222 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-4 (1560382523467366401)]} 0 0
   [junit4]   2> 508222 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-4 (1560382523467366401)]} 0 1
   [junit4]   2> 508257 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-3 (1560382523467366400)]} 0 35
   [junit4]   2> 508259 INFO  (qtp2059649094-26204) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-3 (1560382523467366400)]} 0 37
   [junit4]   2> 508259 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-5 (1560382523503017984)]} 0 4
   [junit4]   2> 508260 INFO  (qtp2059649094-26211) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-5 (1560382523503017984)]} 0 5
   [junit4]   2> 508263 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-2 (-1560382523509309440)]} 0 2
   [junit4]   2> 508263 INFO  (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-4 (1560382523507212288)]} 0 3
   [junit4]   2> 508264 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-4 (1560382523507212288)]} 0 4
   [junit4]   2> 508265 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-2 (-1560382523509309440)]} 0 4
   [junit4]   2> 508266 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-5 (1560382523512455168)]} 0 1
   [junit4]   2> 508267 INFO  (qtp2059649094-26212) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-5 (1560382523512455168)]} 0 2
   [junit4]   2> 508267 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-6 (1560382523513503744)]} 0 0
   [junit4]   2> 508267 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-6 (1560382523513503744)]} 0 2
   [junit4]   2> 508270 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-2 (-1560382523515600896)]} 0 1
   [junit4]   2> 508270 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-7 (1560382523516649472)]} 0 1
   [junit4]   2> 508270 INFO  (qtp2059649094-26204) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-2 (-1560382523515600896)]} 0 2
   [junit4]   2> 508270 INFO  (qtp2059649094-26211) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-7 (1560382523516649472)]} 0 2
   [junit4]   2> 508271 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-6 (1560382523518746624)]} 0 0
   [junit4]   2> 508272 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-6 (1560382523518746624)]} 0 1
   [junit4]   2> 508272 INFO  (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-3 (-1560382523519795200)]} 0 0
   [junit4]   2> 508273 INFO  (qtp2059649094-26211) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-3 (-1560382523519795200)]} 0 2
   [junit4]   2> 508275 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-8 (1560382523522940928)]} 0 1
   [junit4]   2> 508276 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-8 (1560382523522940928)]} 0 2
   [junit4]   2> 508276 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-7 (1560382523520843776)]} 0 0
   [junit4]   2> 508276 INFO  (qtp2059649094-26212) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-7 (1560382523520843776)]} 0 4
   [junit4]   2> 508279 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-5 (-1560382523525038080)]} 0 2
   [junit4]   2> 508280 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-3 (-1560382523527135232)]} 0 1
   [junit4]   2> 508280 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-3 (-1560382523527135232)]} 0 1
   [junit4]   2> 508284 INFO  (qtp2059649094-26204) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-5 (-1560382523525038080)]} 0 8
   [junit4]   2> 508286 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-8 (1560382523534475264)]} 0 0
   [junit4]   2> 508286 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-8 (1560382523534475264)]} 0 1
   [junit4]   2> 508295 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-5 (-1560382523539718144)]} 0 0
   [junit4]   2> 508296 INFO  (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-9 (1560382523535523840)]} 0 3
   [junit4]   2> 508296 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-5 (-1560382523539718144)]} 0 5
   [junit4]   2> 508302 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-9 (1560382523535523840)]} 0 16
   [junit4]   2> 508302 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-9 (1560382523546009600)]} 0 1
   [junit4]   2> 508303 INFO  (qtp2059649094-26212) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-9 (1560382523546009600)]} 0 7
   [junit4]   2> 508304 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-10 (1560382523553349632)]} 0 0
   [junit4]   2> 508304 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-10 (1560382523553349632)]} 0 1
   [junit4]   2> 508306 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-7 (-1560382523555446784)]} 0 0
   [junit4]   2> 508306 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-10 (1560382523554398208)]} 0 0
   [junit4]   2> 508306 INFO  (qtp2059649094-26211) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-7 (-1560382523555446784)]} 0 1
   [junit4]   2> 508306 INFO  (qtp2059649094-26204) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-10 (1560382523554398208)]} 0 2
   [junit4]   2> 508308 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-11 (1560382523557543937)]} 0 0
   [junit4]   2> 508309 INFO  (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-7 (-1560382523557543936)]} 0 0
   [junit4]   2> 508309 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-11 (1560382523557543937)]} 0 1
   [junit4]   2> 508309 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-7 (-1560382523557543936)]} 0 1
   [junit4]   2> 508310 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-11 (-1560382523559641088)]} 0 0
   [junit4]   2> 508311 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-11 (1560382523560689664)]} 0 0
   [junit4]   2> 508311 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-11 (1560382523560689664)]} 0 1
   [junit4]   2> 508311 INFO  (qtp2059649094-26212) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-11 (-1560382523559641088)]} 0 1
   [junit4]   2> 508312 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-12 (1560382523561738240)]} 0 0
   [junit4]   2> 508313 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-11 (-1560382523562786816)]} 0 0
   [junit4]   2> 508313 INFO  (qtp2059649094-26204) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-11 (-1560382523562786816)]} 0 1
   [junit4]   2> 508314 INFO  (qtp2059649094-26211) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-12 (1560382523561738240)]} 0 2
   [junit4]   2> 508314 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-12 (1560382523563835392)]} 0 0
   [junit4]   2> 508314 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-12 (1560382523563835392)]} 0 0
   [junit4]   2> 508316 INFO  (qtp291995094-26197) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-13 (1560382523565932545)]} 0 0
   [junit4]   2> 508316 INFO  (qtp291995094-26203) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-13 (1560382523565932544)]} 0 0
   [junit4]   2> 508316 INFO  (qtp2059649094-26208) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-13 (1560382523565932544)]} 0 1
   [junit4]   2> 508316 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-13 (1560382523565932545)]} 0 1
   [junit4]   2> 508318 INFO  (qtp291995094-26199) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-14 (1560382523568029696)]} 0 0
   [junit4]   2> 508318 INFO  (qtp2059649094-26204) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-14 (1560382523568029696)]} 0 1
   [junit4]   2> 508319 INFO  (qtp291995094-26196) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-12 (-1560382523569078272)]} 0 0
   [junit4]   2> 508319 INFO  (qtp291995094-26201) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-14 (1560382523566981120)]} 0 1
   [junit4]   2> 508319 INFO  (qtp2059649094-26211) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-12 (-1560382523569078272)]} 0 1
   [junit4]   2> 508320 INFO  (qtp2059649094-26212) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-14 (1560382523566981120)]} 0 3
   [junit4]   2> 508321 INFO  (qtp291995094-26202) [n:127.0.0.1:50578_solr c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:46278/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-15 (1560382523571175424)]} 0 0
   [junit4]   2> 508321 INFO  (qtp2059649094-26205) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-15 (1560382523571175424)]} 0 1
   [junit4]   2> 508432 ERROR (updateExecutor-1209-thread-2-processing-http:////127.0.0.1:50578//solr//recoverytest_shard1_replica2 x:recoverytest_shard1_replica1 r:core_node1 n:127.0.0.1:46278_solr s:shard1 c:recoverytest) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.StreamingSolrClients error
   [junit4]   2> org.apache.http.NoHttpResponseException: 127.0.0.1:50578 failed to respond
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2> 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
   [junit4]   2> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
   [junit4]   2> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]   2> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]   2> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]   2> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:371)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:206)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 508451 WARN  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor Error sending update to http://127.0.0.1:50578/solr
   [junit4]   2> org.apache.http.NoHttpResponseException: 127.0.0.1:50578 failed to respond
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2> 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
   [junit4]   2> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
   [junit4]   2> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]   2> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]   2> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]   2> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:371)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:206)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 508446 INFO  (TEST-HdfsRecoveryZkTest.test-seed#[1D48FA3EBA96D4D4]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@765774c9{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 508452 ERROR (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on replica http://127.0.0.1:50578/solr/recoverytest_shard1_replica2/
   [junit4]   2> org.apache.http.NoHttpResponseException: 127.0.0.1:50578 failed to respond
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2> 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:118)
   [junit4]   2> 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
   [junit4]   2> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
   [junit4]   2> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
   [junit4]   2> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
   [junit4]   2> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:371)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:206)
   [junit4]   2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 508455 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.c.ZkController Put replica core=recoverytest_shard1_replica2 coreNodeName=core_node2 on 127.0.0.1:50578_solr into leader-initiated recovery.
   [junit4]   2> 508455 INFO  (qtp2059649094-26210) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-16 (1560382523681275905)]} 0 29
   [junit4]   2> 508445 ERROR (updateExecutor-1209-thread-1-processing-http:////127.0.0.1:50578//solr//recoverytest_shard1_replica2 x:recoverytest_shard1_replica1 r:core_node1 n:127.0.0.1:46278_solr s:shard1 c:recoverytest) [n:127.0.0.1:46278_solr c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] o.a.s.u.StreamingSolrClients error
   [junit4]   2> org.apache.http.NoHttpResponseException: 127.0.0.1:50578 failed to respond
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]   2> 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2> 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2> 	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
   [junit4]   2> 	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
   [junit4]   2> 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveRes

[...truncated too long message...]

eanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:204)
   [junit4]   2> 	at org.apache.solr.core.HdfsDirectoryFactoryTest.teardownClass(HdfsDirectoryFactoryTest.java:61)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> 	... 53 more
   [junit4]   2> 3334702 INFO  (SUITE-HdfsDirectoryFactoryTest-seed#[1D48FA3EBA96D4D4]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.core.HdfsDirectoryFactoryTest_1D48FA3EBA96D4D4-001
   [junit4]   2> Feb 26, 2017 8:59:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=id, timezone=UTC
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=328025448,total=523239424
   [junit4]   2> NOTE: All tests run in this JVM: [TestSSLRandomization, LukeRequestHandlerTest, HardAutoCommitTest, WrapperMergePolicyFactoryTest, TestIndexingPerformance, DocExpirationUpdateProcessorFactoryTest, RequestHandlersTest, RecoveryZkTest, TestDistributedMissingSort, AddSchemaFieldsUpdateProcessorFactoryTest, DocValuesNotIndexedTest, DebugComponentTest, HdfsSyncSliceTest, TestClusterStateMutator, DateFieldTest, HdfsDirectoryTest, NoCacheHeaderTest, TestDFISimilarityFactory, FullSolrCloudDistribCmdsTest, TestFieldCache, DistanceUnitsTest, SuggestComponentTest, JvmMetricsTest, SynonymTokenizerTest, TestLocalFSCloudBackupRestore, OverriddenZkACLAndCredentialsProvidersTest, MBeansHandlerTest, BooleanFieldTest, TestFilteredDocIdSet, V2ApiIntegrationTest, TestSchemalessBufferedUpdates, TestRestoreCore, TestQueryWrapperFilter, TestSolrQueryParser, DistribJoinFromCollectionTest, TestSort, AutoCommitTest, TestFastOutputStream, TestFieldCacheWithThreads, TestSolr4Spatial2, TestComplexPhraseQParserPlugin, TestShardHandlerFactory, FieldAnalysisRequestHandlerTest, MissingSegmentRecoveryTest, TestFieldTypeResource, TestSizeLimitedDistributedMap, TestLuceneMatchVersion, PeerSyncWithIndexFingerprintCachingTest, TestRTimerTree, TestBulkSchemaAPI, BaseCdcrDistributedZkTest, TestApiFramework, TestJsonFacets, TestCryptoKeys, TestSolrCloudWithHadoopAuthPlugin, TestExtendedDismaxParser, TestPivotHelperCode, StatelessScriptUpdateProcessorFactoryTest, TestBinaryField, TestConfigSets, SolrCoreCheckLockOnStartupTest, RAMDirectoryFactoryTest, TestReload, TestSchemaSimilarityResource, HighlighterConfigTest, TestGroupingSearch, TestNamedUpdateProcessors, RequestLoggingTest, TestRecovery, LeaderElectionContextKeyTest, UtilsToolTest, BasicAuthIntegrationTest, TestManagedResource, TestSortByMinMaxFunction, TlogReplayBufferedWhileIndexingTest, TestExceedMaxTermLength, CustomCollectionTest, SolrGraphiteReporterTest, TestDynamicLoading, TestZkChroot, OutputWriterTest, TestJavabinTupleStreamParser, TestDocSet, TestExportWriter, TestCSVResponseWriter, TestLMDirichletSimilarityFactory, TestConfigOverlay, TestCloudManagedSchema, HighlighterMaxOffsetTest, CoreAdminHandlerTest, CdcrReplicationHandlerTest, TestPHPSerializedResponseWriter, TestExactSharedStatsCache, TestDefaultSearchFieldResource, TestSolrDeletionPolicy1, AnalysisErrorHandlingTest, TestSolrConfigHandlerCloud, TestFieldCacheSort, CoreMergeIndexesAdminHandlerTest, SuggesterTSTTest, TestCloudRecovery, CloudMLTQParserTest, SolrXmlInZkTest, SolrTestCaseJ4Test, TestStressLiveNodes, CurrencyFieldXmlFileTest, MetricUtilsTest, XmlUpdateRequestHandlerTest, TestUtils, ConfigSetsAPITest, SuggesterTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OverseerTest, TestRandomFaceting, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, CreateCollectionCleanupTest, DeleteNodeTest, DeleteStatusTest, ForceLeaderTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerRolesTest, OverseerStatusTest, PeerSyncReplicationTest, RecoveryAfterSoftCommitTest, ReplicationFactorTest, RestartWhileUpdatingTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestClusterProperties, TestConfigSetsAPI, TestDownShardTolerantSearch, TestLeaderElectionWithEmptyReplica, TestRandomRequestDistribution, TestReplicaProperties, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressInPlaceUpdates, HdfsChaosMonkeyNothingIsSafeTest, HdfsChaosMonkeySafeLeaderTest, HdfsRecoverLeaseTest, HdfsThreadLeakTest, HdfsUnloadDistributedZkTest, ZkStateReaderTest, ZkStateWriterTest, RuleEngineTest, BlobRepositoryCloudTest, CoreSorterTest, HdfsDirectoryFactoryTest]
   [junit4] Completed [557/693 (2!)] on J2 in 13.40s, 3 tests, 1 error <<< FAILURES!

[...truncated 63062 lines...]



[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1247 - Failure

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1247/

5 tests failed.
FAILED:  org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:76)
	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:129)
	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:35)
	at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:979)
	at org.apache.lucene.util.fst.NodeHash.hash(NodeHash.java:111)
	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:128)
	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
	at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
	at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finishTerm(MemoryPostingsFormat.java:245)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.access$300(MemoryPostingsFormat.java:104)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:386)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
	at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:109)
	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:163)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:444)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539)
	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:396)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:499)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1579)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1324)
	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
	at org.apache.lucene.index.TestDuelingCodecs.createRandomIndex(TestDuelingCodecs.java:139)
	at org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals(TestDuelingCodecsAtNight.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)


FAILED:  org.apache.lucene.search.TestFuzzyQuery.testRandom

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestFuzzyQuery

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)


FAILED:  org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery

Error Message:
Expected a collection with one shard and two replicas null Last available state: DocCollection(MissingSegmentRecoveryTest//collections/MissingSegmentRecoveryTest/state.json/7)={   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"MissingSegmentRecoveryTest_shard1_replica2",           "base_url":"http://127.0.0.1:53224/solr",           "node_name":"127.0.0.1:53224_solr",           "state":"active",           "leader":"true"},         "core_node2":{           "core":"MissingSegmentRecoveryTest_shard1_replica1",           "base_url":"http://127.0.0.1:44586/solr",           "node_name":"127.0.0.1:44586_solr",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected a collection with one shard and two replicas
null
Last available state: DocCollection(MissingSegmentRecoveryTest//collections/MissingSegmentRecoveryTest/state.json/7)={
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"MissingSegmentRecoveryTest_shard1_replica2",
          "base_url":"http://127.0.0.1:53224/solr",
          "node_name":"127.0.0.1:53224_solr",
          "state":"active",
          "leader":"true"},
        "core_node2":{
          "core":"MissingSegmentRecoveryTest_shard1_replica1",
          "base_url":"http://127.0.0.1:44586/solr",
          "node_name":"127.0.0.1:44586_solr",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
	at __randomizedtesting.SeedInfo.seed([24476FD2E1B70732:7412F7D1B896B12F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:265)
	at org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery(MissingSegmentRecoveryTest.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:58627/ies: Error CREATEing SolrCore 'test_unload_shard_and_collection_1': Unable to create core [test_unload_shard_and_collection_1] Caused by: Direct buffer memory

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:58627/ies: Error CREATEing SolrCore 'test_unload_shard_and_collection_1': Unable to create core [test_unload_shard_and_collection_1] Caused by: Direct buffer memory
	at __randomizedtesting.SeedInfo.seed([24476FD2E1B70732:AC1350084F4B6ACA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:125)
	at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 736 lines...]
   [junit4] Suite: org.apache.lucene.index.TestDuelingCodecsAtNight
   [junit4]   2> Shk 25, 2017 1:28:47 MD com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #30,5,TGRP-TestDuelingCodecsAtNight]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 	at org.apache.lucene.util.fst.PairOutputs.newPair(PairOutputs.java:90)
   [junit4]   2> 	at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:140)
   [junit4]   2> 	at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:32)
   [junit4]   2> 	at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
   [junit4]   2> 	at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
   [junit4]   2> 	at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
   [junit4]   2> 	at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:170)
   [junit4]   2> 	at org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:275)
   [junit4]   2> 	at org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:301)
   [junit4]   2> 	at org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:196)
   [junit4]   2> 	at org.apache.lucene.codecs.simpletext.SimpleTextFieldsWriter.write(SimpleTextFieldsWriter.java:101)
   [junit4]   2> 	at org.apache.lucene.codecs.simpletext.SimpleTextFieldsWriter.write(SimpleTextFieldsWriter.java:60)
   [junit4]   2> 	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4338)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3915)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]   2> 
   [junit4]   2> Shk 25, 2017 1:28:47 MD com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #4,5,TGRP-TestDuelingCodecsAtNight]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:129)
   [junit4]   2> 	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:35)
   [junit4]   2> 	at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:979)
   [junit4]   2> 	at org.apache.lucene.util.fst.NodeHash.hash(NodeHash.java:111)
   [junit4]   2> 	at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:148)
   [junit4]   2> 	at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   [junit4]   2> 	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   [junit4]   2> 	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]   2> 	at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
   [junit4]   2> 	at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
   [junit4]   2> 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finishTerm(MemoryPostingsFormat.java:245)
   [junit4]   2> 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.access$300(MemoryPostingsFormat.java:104)
   [junit4]   2> 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:386)
   [junit4]   2> 	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]   2> 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4338)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3915)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]   2> 
   [junit4]   2> Shk 25, 2017 1:28:47 MD com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #2,5,TGRP-TestDuelingCodecsAtNight]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 	at org.apache.lucene.util.fst.BytesStore.skipBytes(BytesStore.java:307)
   [junit4]   2> 	at org.apache.lucene.util.fst.FST.addNode(FST.java:707)
   [junit4]   2> 	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:126)
   [junit4]   2> 	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]   2> 	at org.apache.lucene.util.fst.Builder.finish(Builder.java:483)
   [junit4]   2> 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finish(MemoryPostingsFormat.java:258)
   [junit4]   2> 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:391)
   [junit4]   2> 	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]   2> 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]   2> 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4338)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3915)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDuelingCodecsAtNight -Dtests.method=testBigEquals -Dtests.seed=2279865B45AB59A9 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=sq-AL -Dtests.timezone=Africa/Nairobi -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    357s J0 | TestDuelingCodecsAtNight.testBigEquals <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at org.apache.lucene.util.BytesRef.<init>(BytesRef.java:76)
   [junit4]    > 	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:129)
   [junit4]    > 	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:35)
   [junit4]    > 	at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:979)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.hash(NodeHash.java:111)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:128)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finishTerm(MemoryPostingsFormat.java:245)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.access$300(MemoryPostingsFormat.java:104)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:386)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:140)
   [junit4]    > 	at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:163)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:444)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:396)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:499)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1579)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1324)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:171)
   [junit4]    > 	at org.apache.lucene.index.TestDuelingCodecs.createRandomIndex(TestDuelingCodecs.java:139)
   [junit4]    > 	at org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals(TestDuelingCodecsAtNight.java:34)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3853, name=Lucene Merge Thread #30, state=RUNNABLE, group=TGRP-TestDuelingCodecsAtNight]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at org.apache.lucene.util.fst.PairOutputs.newPair(PairOutputs.java:90)
   [junit4]    > 	at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:140)
   [junit4]    > 	at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:32)
   [junit4]    > 	at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
   [junit4]    > 	at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
   [junit4]    > 	at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
   [junit4]    > 	at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:170)
   [junit4]    > 	at org.apache.lucene.index.MultiTermsEnum.pushTop(MultiTermsEnum.java:275)
   [junit4]    > 	at org.apache.lucene.index.MultiTermsEnum.next(MultiTermsEnum.java:301)
   [junit4]    > 	at org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:196)
   [junit4]    > 	at org.apache.lucene.codecs.simpletext.SimpleTextFieldsWriter.write(SimpleTextFieldsWriter.java:101)
   [junit4]    > 	at org.apache.lucene.codecs.simpletext.SimpleTextFieldsWriter.write(SimpleTextFieldsWriter.java:60)
   [junit4]    > 	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4338)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3915)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3856, name=Lucene Merge Thread #2, state=RUNNABLE, group=TGRP-TestDuelingCodecsAtNight]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at org.apache.lucene.util.fst.BytesStore.skipBytes(BytesStore.java:307)
   [junit4]    > 	at org.apache.lucene.util.fst.FST.addNode(FST.java:707)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:126)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.finish(Builder.java:483)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finish(MemoryPostingsFormat.java:258)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:391)
   [junit4]    > 	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4338)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3915)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3858, name=Lucene Merge Thread #4, state=RUNNABLE, group=TGRP-TestDuelingCodecsAtNight]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:129)
   [junit4]    > 	at org.apache.lucene.util.fst.ByteSequenceOutputs.read(ByteSequenceOutputs.java:35)
   [junit4]    > 	at org.apache.lucene.util.fst.FST.readNextRealArc(FST.java:979)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.hash(NodeHash.java:111)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.addNew(NodeHash.java:148)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.rehash(NodeHash.java:169)
   [junit4]    > 	at org.apache.lucene.util.fst.NodeHash.add(NodeHash.java:133)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.compileNode(Builder.java:200)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:296)
   [junit4]    > 	at org.apache.lucene.util.fst.Builder.add(Builder.java:400)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.finishTerm(MemoryPostingsFormat.java:245)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsWriter.access$300(MemoryPostingsFormat.java:104)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$MemoryFieldsConsumer.write(MemoryPostingsFormat.java:386)
   [junit4]    > 	at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4338)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3915)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J0/temp/lucene.index.TestDuelingCodecsAtNight_2279865B45AB59A9-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {titleTokenized=DFR GB2, body=DFR I(ne)B1}, locale=sq-AL, timezone=Africa/Nairobi
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=99384176,total=411566080
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterForceMerge, TestByteArrayDataInput, TestDocValuesIndexing, TestPersistentSnapshotDeletionPolicy, TestSloppyPhraseQuery, TestIndexableField, TestMultiPhraseEnum, TestPagedBytes, TestDirectoryReaderReopen, TestPointQueries, TestTotalHitCountCollector, TestIndexWriterConfig, TestMatchAllDocsQuery, TestSingleInstanceLockFactory, TestSimilarityProvider, TestBytesStore, TestLucene50TermVectorsFormat, TestNRTCachingDirectory, TestTermsEnum, TestDocsAndPositions, TestAutomatonQuery, TestIndexWriterExceptions2, Test4GBStoredFields, TestOrdinalMap, TestDeterminizeLexicon, TestTimSorter, TestMultiPhraseQuery, TestTopFieldCollector, TestSimpleExplanations, TestTermVectorsWriter, TestUniqueTermCount, TestThreadedForceMerge, TestDateTools, TestUnicodeUtil, TestIndexInput, TestLiveFieldValues, TestNeverDelete, TestBoolean2, TestRegExp, TestApproximationSearchEquivalence, Test2BBinaryDocValues, TestSort, TestLucene62SegmentInfoFormat, TestDuelingCodecsAtNight]
   [junit4] Completed [154/448 (1!)] on J0 in 381.03s, 3 tests, 1 error <<< FAILURES!

[...truncated 1132 lines...]
   [junit4] Suite: org.apache.lucene.search.TestFuzzyQuery
   [junit4]   2> f?vr. 25, 2017 8:47:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.search.TestFuzzyQuery
   [junit4]   2>    1) Thread[id=7378, name=SUITE-TestFuzzyQuery-seed#[2279865B45AB59A9], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1556)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:696)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:693)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:693)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:709)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:689)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:678)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:598)
   [junit4]   2>    2) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1249)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1323)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:608)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:457)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2>    3) Thread[id=7379, name=TEST-TestFuzzyQuery.testRandom-seed#[2279865B45AB59A9], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    4) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery -Dtests.method=testRandom -Dtests.seed=2279865B45AB59A9 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=fr-CA -Dtests.timezone=America/Porto_Velho -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   7200s J0 | TestFuzzyQuery.testRandom <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4]   2> f?vr. 25, 2017 8:47:38 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> f?vr. 25, 2017 8:47:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.search.TestFuzzyQuery: 
   [junit4]   2>    1) Thread[id=7379, name=TEST-TestFuzzyQuery.testRandom-seed#[2279865B45AB59A9], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at java.util.WeakHashMap.getTable(WeakHashMap.java:350)
   [junit4]   2>         at java.util.WeakHashMap.getEntry(WeakHashMap.java:427)
   [junit4]   2>         at java.util.WeakHashMap.containsKey(WeakHashMap.java:417)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:258)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
   [junit4]   2>         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:506)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> f?vr. 25, 2017 8:47:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=7379, name=TEST-TestFuzzyQuery.testRandom-seed#[2279865B45AB59A9], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2> f?vr. 25, 2017 8:48:02 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=7379, name=TEST-TestFuzzyQuery.testRandom-seed#[2279865B45AB59A9], state=RUNNABLE, group=TGRP-TestFuzzyQuery]
   [junit4]   2>         at java.lang.Object.hashCode(Native Method)
   [junit4]   2>         at java.util.WeakHashMap.hash(WeakHashMap.java:298)
   [junit4]   2>         at java.util.WeakHashMap.get(WeakHashMap.java:396)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.getPerThread(RandomizedContext.java:319)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandomness(RandomizedContext.java:92)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedContext.getRandom(RandomizedContext.java:114)
   [junit4]   2>         at org.apache.lucene.util.LuceneTestCase.random(LuceneTestCase.java:748)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.randomSimpleString(TestFuzzyQuery.java:503)
   [junit4]   2>         at org.apache.lucene.search.TestFuzzyQuery.testRandom(TestFuzzyQuery.java:517)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=fr-CA, timezone=America/Porto_Velho
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=2,free=165761096,total=411041792
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterForceMerge, TestByteArrayDataInput, TestDocValuesIndexing, TestPersistentSnapshotDeletionPolicy, TestSloppyPhraseQuery, TestIndexableField, TestMultiPhraseEnum, TestPagedBytes, TestDirectoryReaderReopen, TestPointQueries, TestTotalHitCountCollector, TestIndexWriterConfig, TestMatchAllDocsQuery, TestSingleInstanceLockFactory, TestSimilarityProvider, TestBytesStore, TestLucene50TermVectorsFormat, TestNRTCachingDirectory, TestTermsEnum, TestDocsAndPositions, TestAutomatonQuery, TestIndexWriterExceptions2, Test4GBStoredFields, TestOrdinalMap, TestDeterminizeLexicon, TestTimSorter, TestMultiPhraseQuery, TestTopFieldCollector, TestSimpleExplanations, TestTermVectorsWriter, TestUniqueTermCount, TestThreadedForceMerge, TestDateTools, TestUnicodeUtil, TestIndexInput, TestLiveFieldValues, TestNeverDelete, TestBoolean2, TestRegExp, TestApproximationSearchEquivalence, Test2BBinaryDocValues, TestSort, TestLucene62SegmentInfoFormat, TestDuelingCodecsAtNight, TestUTF32ToUTF8, TestFixedBitDocIdSet, TestDocInverterPerFieldErrorInfo, TestCrashCausesCorruptIndex, TestSpanOrQuery, Test2BPoints, TestFilterSpans, TestLucene70NormsFormat, TestIndexingSequenceNumbers, TestBooleanQueryVisitSubscorers, TestNotDocIdSet, TestFlex, TestNearSpansOrdered, TestDeletionPolicy, TestLucene50CompoundFormat, TestIndexWriterExceptions, TestMatchNoDocsQuery, TestIndexSearcher, TestRateLimiter, TestQueryBuilder, TestSpanContainQuery, TestFieldReuse, TestMultiThreadTermVectors, TestBytesRef, TestForceMergeForever, TestCodecs, TestIndexWriter, TestInPlaceMergeSorter, TestStressDeletes, TestBooleanRewrites, TestTopDocsMerge, TestSnapshotDeletionPolicy, TestNRTReaderWithThreads, TestPerFieldPostingsFormat, TestDirectoryReader, TestLockFactory, TestSegmentMerger, TestCollectionUtil, TestOmitNorms, TestFixedBitSet, TestTransactions, TestCrash, TestSegmentTermDocs, TestMultiDocValues, TestToken, TestFieldsReader, TestSimpleSearchEquivalence, TestCustomSearcherSort, TestHugeRamFile, TestBooleanMinShouldMatch, TestTermsEnum2, TestIndexWriterOnDiskFull, TestFuzzyQuery]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFuzzyQuery -Dtests.seed=2279865B45AB59A9 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=fr-CA -Dtests.timezone=America/Porto_Velho -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | TestFuzzyQuery (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2279865B45AB59A9]:0)
   [junit4] Completed [448/448 (2!)] on J0 in 7223.09s, 2 tests, 2 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/temp/junit4-J0-20170225_100833_1583846647856942843690.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid23943.hprof ...
   [junit4] Heap dump file created [430529117 bytes in 0.628 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9971 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/init-core-data-001
   [junit4]   2> 613761 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 613762 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 613762 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ies/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 613802 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 613810 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 613812 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 613825 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_55473_hdfs____w63yu2/webapp
   [junit4]   2> 614218 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55473
   [junit4]   2> 614379 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 614380 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 614390 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_58748_datanode____.slbj5/webapp
   [junit4]   2> 614891 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:58748
   [junit4]   2> 614973 WARN  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 614974 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 614983 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_56221_datanode____d5fp93/webapp
   [junit4]   2> 615128 INFO  (IPC Server handler 3 on 56059) [    ] BlockStateChange BLOCK* processReport: from storage DS-e6ac0cef-8c53-4ad0-9d39-4343fdd5924d node DatanodeRegistration(127.0.0.1:33878, datanodeUuid=b44b3547-77ee-4ecb-8f30-f4c02396e694, infoPort=34919, infoSecurePort=0, ipcPort=56414, storageInfo=lv=-56;cid=testClusterID;nsid=275081903;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 615128 INFO  (IPC Server handler 3 on 56059) [    ] BlockStateChange BLOCK* processReport: from storage DS-37f14335-7ad4-48b4-a6ba-3bccd1ce3697 node DatanodeRegistration(127.0.0.1:33878, datanodeUuid=b44b3547-77ee-4ecb-8f30-f4c02396e694, infoPort=34919, infoSecurePort=0, ipcPort=56414, storageInfo=lv=-56;cid=testClusterID;nsid=275081903;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 615405 INFO  (SUITE-HdfsUnloadDistributedZkTest-seed#[24476FD2E1B70732]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56221
   [junit4]   2> 615574 INFO  (IPC Server handler 5 on 56059) [    ] BlockStateChange BLOCK* processReport: from storage DS-cb1b130e-c867-440b-9fec-844fc0b08b10 node DatanodeRegistration(127.0.0.1:59880, datanodeUuid=aa00d148-acae-4d18-abf7-f13cd248efa3, infoPort=59421, infoSecurePort=0, ipcPort=59132, storageInfo=lv=-56;cid=testClusterID;nsid=275081903;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 615574 INFO  (IPC Server handler 5 on 56059) [    ] BlockStateChange BLOCK* processReport: from storage DS-e39c9cb9-f572-431f-a821-eb6a22637b42 node DatanodeRegistration(127.0.0.1:59880, datanodeUuid=aa00d148-acae-4d18-abf7-f13cd248efa3, infoPort=59421, infoSecurePort=0, ipcPort=59132, storageInfo=lv=-56;cid=testClusterID;nsid=275081903;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 615816 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 615816 INFO  (Thread-1139) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 615816 INFO  (Thread-1139) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 615916 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.ZkTestServer start zk server on port:45178
   [junit4]   2> 615933 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 615934 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 615935 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 615936 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 615937 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 615938 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 615939 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 615939 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 615940 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 615941 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 615942 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 616079 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/control-001/cores/collection1
   [junit4]   2> 616081 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 616084 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1cf14788{/ies,null,AVAILABLE}
   [junit4]   2> 616084 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7d92a476{HTTP/1.1,[http/1.1]}{127.0.0.1:45074}
   [junit4]   2> 616084 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.Server Started @619359ms
   [junit4]   2> 616084 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:56059/hdfs__localhost_56059__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001_tempDir-002_control_data, hostContext=/ies, hostPort=45074, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/control-001/cores}
   [junit4]   2> 616085 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 616085 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.0.0
   [junit4]   2> 616085 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 616085 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 616085 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-25T14:46:20.752Z
   [junit4]   2> 616089 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 616089 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/control-001/solr.xml
   [junit4]   2> 616095 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45178/solr
   [junit4]   2> 616121 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [n:127.0.0.1:45074_ies    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 616122 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [n:127.0.0.1:45074_ies    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45074_ies
   [junit4]   2> 616122 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [n:127.0.0.1:45074_ies    ] o.a.s.c.Overseer Overseer (id=97519794944999428-127.0.0.1:45074_ies-n_0000000000) starting
   [junit4]   2> 616137 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [n:127.0.0.1:45074_ies    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45074_ies
   [junit4]   2> 616138 INFO  (zkCallback-2376-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 616256 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [n:127.0.0.1:45074_ies    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/control-001/cores
   [junit4]   2> 616256 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [n:127.0.0.1:45074_ies    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 616271 INFO  (OverseerStateUpdate-97519794944999428-127.0.0.1:45074_ies-n_0000000000) [n:127.0.0.1:45074_ies    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 617273 WARN  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 617274 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 617285 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 617371 WARN  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 617373 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 617398 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 617410 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:56059/solr_hdfs_home
   [junit4]   2> 617410 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 617411 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/control-001/cores/collection1], dataDir=[hdfs://localhost:56059/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 617411 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3202fcd3
   [junit4]   2> 617411 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:56059/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 617426 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 617426 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 617426 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 617914 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 617919 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:56059/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 617945 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:56059/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 617951 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 617951 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 617951 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 617963 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 617964 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=6, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5078474487904397]
   [junit4]   2> 618006 INFO  (IPC Server handler 8 on 56059) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33878 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-cb1b130e-c867-440b-9fec-844fc0b08b10:NORMAL:127.0.0.1:59880|RBW], ReplicaUC[[DISK]DS-37f14335-7ad4-48b4-a6ba-3bccd1ce3697:NORMAL:127.0.0.1:33878|FINALIZED]]} size 0
   [junit4]   2> 618006 INFO  (IPC Server handler 9 on 56059) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59880 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-37f14335-7ad4-48b4-a6ba-3bccd1ce3697:NORMAL:127.0.0.1:33878|FINALIZED], ReplicaUC[[DISK]DS-e39c9cb9-f572-431f-a821-eb6a22637b42:NORMAL:127.0.0.1:59880|FINALIZED]]} size 0
   [junit4]   2> 618031 WARN  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 618067 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 618067 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 618067 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 618077 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 618077 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 618079 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=30.365234375, floorSegmentMB=0.435546875, forceMergeDeletesPctAllowed=16.399156366641176, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 618154 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3bb375e0[collection1] main]
   [junit4]   2> 618155 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 618156 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 618158 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 618162 INFO  (searcherExecutor-2337-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3bb375e0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 618175 INFO  (coreLoadExecutor-2336-thread-1-processing-n:127.0.0.1:45074_ies) [n:127.0.0.1:45074_ies c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560316721592532992
   [junit4]   2> 618184 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 618184 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 618184 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45074/ies/collection1/
   [junit4]   2> 618184 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 618184 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:45074/ies/collection1/ has no replicas
   [junit4]   2> 618185 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 618187 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45074/ies/collection1/ shard1
   [junit4]   2> 618337 INFO  (coreZkRegister-2329-thread-1-processing-n:127.0.0.1:45074_ies x:collection1 c:control_collection) [n:127.0.0.1:45074_ies c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 618394 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 618395 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45178/solr ready
   [junit4]   2> 618395 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 618395 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 618486 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/shard-1-001/cores/collection1
   [junit4]   2> 618486 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/shard-1-001
   [junit4]   2> 618487 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 618488 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1b6bb0a6{/ies,null,AVAILABLE}
   [junit4]   2> 618488 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7cddd679{HTTP/1.1,[http/1.1]}{127.0.0.1:58627}
   [junit4]   2> 618493 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.e.j.s.Server Started @621768ms
   [junit4]   2> 618493 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:56059/hdfs__localhost_56059__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-master_checkout_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/ies, hostPort=58627, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsUnloadDistributedZkTest_24476FD2E1B70732-001/shard-1-001/cores}
   [junit4]   2> 618493 ERROR (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 618494 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solrâ„¢ version 7.0.0
   [junit4]   2> 618494 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 618494 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 618494 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-25T14:46:23.161Z
   [junit4]   2> 618496 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#[24476FD2E1B70732]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 618496 INFO  (TEST-HdfsUnloadDistributedZkTest.test-seed#

[...truncated too long message...]

ction(MissingSegmentRecoveryTest//collections/MissingSegmentRecoveryTest/state.json/7)={
   [junit4]    >   "replicationFactor":"2",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{
   [junit4]    >         "core_node1":{
   [junit4]    >           "core":"MissingSegmentRecoveryTest_shard1_replica2",
   [junit4]    >           "base_url":"http://127.0.0.1:53224/solr",
   [junit4]    >           "node_name":"127.0.0.1:53224_solr",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"},
   [junit4]    >         "core_node2":{
   [junit4]    >           "core":"MissingSegmentRecoveryTest_shard1_replica1",
   [junit4]    >           "base_url":"http://127.0.0.1:44586/solr",
   [junit4]    >           "node_name":"127.0.0.1:44586_solr",
   [junit4]    >           "state":"down"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false"}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([24476FD2E1B70732:7412F7D1B896B12F]:0)
   [junit4]    > 	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:265)
   [junit4]    > 	at org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery(MissingSegmentRecoveryTest.java:105)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2647514 INFO  (jetty-closer-1819-thread-2) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@2d3f734a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2647515 INFO  (jetty-closer-1819-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1057763130
   [junit4]   2> 2647520 INFO  (jetty-closer-1819-thread-2) [    ] o.a.s.c.Overseer Overseer (id=97519921748246534-127.0.0.1:53224_solr-n_0000000000) closing
   [junit4]   2> 2647520 INFO  (jetty-closer-1819-thread-1) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@3f28cc19{HTTP/1.1,[http/1.1]}{127.0.0.1:44586}
   [junit4]   2> 2647520 INFO  (OverseerStateUpdate-97519921748246534-127.0.0.1:53224_solr-n_0000000000) [n:127.0.0.1:53224_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:53224_solr
   [junit4]   2> 2647522 INFO  (jetty-closer-1819-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1432069275
   [junit4]   2> 2647523 INFO  (zkCallback-1841-thread-3-processing-n:127.0.0.1:44586_solr) [n:127.0.0.1:44586_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44586_solr
   [junit4]   2> 2647524 INFO  (zkCallback-1841-thread-4-processing-n:127.0.0.1:44586_solr) [n:127.0.0.1:44586_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 2649023 WARN  (zkCallback-1830-thread-16-processing-n:127.0.0.1:53224_solr) [n:127.0.0.1:53224_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2649023 INFO  (jetty-closer-1819-thread-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2649024 INFO  (jetty-closer-1819-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@134e640d{/solr,null,UNAVAILABLE}
   [junit4]   2> 2649032 WARN  (zkCallback-1841-thread-4-processing-n:127.0.0.1:44586_solr) [n:127.0.0.1:44586_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2649032 INFO  (jetty-closer-1819-thread-1) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2649033 INFO  (jetty-closer-1819-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6fa11df1{/solr,null,UNAVAILABLE}
   [junit4]   2> 2649034 INFO  (SUITE-MissingSegmentRecoveryTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53711 53711
   [junit4]   2> 2649122 INFO  (Thread-21056) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53711 53711
   [junit4]   2> 2649123 WARN  (Thread-21056) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 	4	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 	2	/solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/MissingSegmentRecoveryTest/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/97519921748246534-127.0.0.1:53224_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	93	/solr/overseer/collection-queue-work
   [junit4]   2> 	13	/solr/overseer/queue
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 	4	/solr/overseer/queue-work
   [junit4]   2> 	3	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 2649123 INFO  (SUITE-MissingSegmentRecoveryTest-seed#[24476FD2E1B70732]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.MissingSegmentRecoveryTest_24476FD2E1B70732-001
   [junit4]   2> Feb 25, 2017 3:20:13 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {_version_=PostingsFormat(name=Memory), id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, maxPointsInLeafNode=1550, maxMBSortInHeap=6.71678823894384, sim=RandomSimilarity(queryNorm=false): {}, locale=el-GR, timezone=America/Argentina/Catamarca
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=228711456,total=511705088
   [junit4]   2> NOTE: All tests run in this JVM: [TestJsonRequest, FullHLLTest, TestBM25SimilarityFactory, BufferStoreTest, TestPerFieldSimilarity, RemoteQueryErrorTest, BlockJoinFacetDistribTest, TestTrie, TestCustomDocTransformer, ExitableDirectoryReaderTest, DocumentBuilderTest, StressHdfsTest, TestConfigSetProperties, CdcrVersionReplicationTest, TestSweetSpotSimilarityFactory, SolrInfoMBeanTest, TestDocBasedVersionConstraints, TestMiniSolrCloudCluster, TestInPlaceUpdatesDistrib, TestFieldTypeCollectionResource, SoftAutoCommitTest, TestIntervalFaceting, TestSystemCollAutoCreate, UUIDUpdateProcessorFallbackTest, DistributedMLTComponentTest, TestCustomStream, TestBackupRepositoryFactory, ClusterStateUpdateTest, TestCollectionAPI, ParsingFieldUpdateProcessorsTest, TestStressRecovery, TestObjectReleaseTracker, TestFiltering, TestFieldCacheSanityChecker, ResponseHeaderTest, TestExpandComponent, JavabinLoaderTest, TestEmbeddedSolrServerConstructors, TestTolerantUpdateProcessorRandomCloud, TestHighFrequencyDictionaryFactory, TestConfigSetsAPIZkFailure, CloneFieldUpdateProcessorFactoryTest, SpatialFilterTest, TestSchemaManager, JsonLoaderTest, HttpPartitionTest, TestFastLRUCache, TestWordDelimiterFilterFactory, MetricsHandlerTest, TestCopyFieldCollectionResource, ZkCLITest, TemplateUpdateProcessorTest, ClassificationUpdateProcessorFactoryTest, DocValuesTest, ChaosMonkeyNothingIsSafeTest, SOLR749Test, TestCloudDeleteByQuery, SortSpecParsingTest, HdfsChaosMonkeyNothingIsSafeTest, TestReqParamsAPI, TestSolrCoreSnapshots, TestQueryUtils, DistributedSuggestComponentTest, TestReloadDeadlock, StatsReloadRaceTest, HdfsTlogReplayBufferedWhileIndexingTest, FileUtilsTest, VersionInfoTest, TestConfigsApi, SchemaVersionSpecificBehaviorTest, JSONWriterTest, SharedFSAutoReplicaFailoverUtilsTest, TestDocumentBuilder, RulesTest, BigEndianAscendingWordSerializerTest, RestartWhileUpdatingTest, TestReloadAndDeleteDocs, DateRangeFieldTest, TestPerFieldSimilarityWithDefaultOverride, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, TestDynamicFieldResource, CachingDirectoryFactoryTest, ReplicaListTransformerTest, TestOnReconnectListenerSupport, TimeZoneUtilsTest, CollectionsAPIDistributedZkTest, HdfsNNFailoverTest, SortByFunctionTest, TestPostingsSolrHighlighter, TestSlowCompositeReaderWrapper, DistributedExpandComponentTest, IndexSchemaRuntimeFieldTest, SparseHLLTest, CreateCollectionCleanupTest, HighlighterTest, DistributedTermsComponentTest, ActionThrottleTest, LeaderFailureAfterFreshStartTest, WordBreakSolrSpellCheckerTest, TestSmileRequest, OverseerTaskQueueTest, TestOverriddenPrefixQueryForCustomFieldType, TestChildDocTransformer, MigrateRouteKeyTest, SpellCheckComponentTest, TestSQLHandlerNonCloud, QueryParsingTest, HdfsRestartWhileUpdatingTest, TestManagedSchemaAPI, DistributedIntervalFacetingTest, OpenExchangeRatesOrgProviderTest, ConnectionManagerTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteShardTest, DeleteStatusTest, DistribDocExpirationUpdateProcessorTest, LeaderElectionContextKeyTest, MissingSegmentRecoveryTest]
   [junit4] Completed [516/693 (2!)] on J1 in 98.87s, 1 test, 1 failure <<< FAILURES!

[...truncated 12297 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:783: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:727: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid23943.hprof

Total time: 345 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1246 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1246/

2 tests failed.
FAILED:  org.apache.lucene.search.TestShardSearching.testSimple

Error Message:
wrong hit docID expected:<11> but was:<3>

Stack Trace:
java.lang.AssertionError: wrong hit docID expected:<11> but was:<3>
	at __randomizedtesting.SeedInfo.seed([D94E59EA03A58262:E1FD7D14245656B3]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.lucene.util.TestUtil.assertEquals(TestUtil.java:1050)
	at org.apache.lucene.search.TestShardSearching.assertSame(TestShardSearching.java:387)
	at org.apache.lucene.search.TestShardSearching.testSimple(TestShardSearching.java:236)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.update.TestInPlaceUpdatesDistrib.test

Error Message:
'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@5c6917db (not leader) wrong [docid] for SolrDocument{id=0, id_field_copy_that_does_not_support_in_place_update_s=0, title_s=title0, id_i=0, inplace_updatable_float=101.0, _version_=1560229605069029376, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=9487} expected:<7129> but was:<9487>

Stack Trace:
java.lang.AssertionError: 'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@5c6917db (not leader) wrong [docid] for SolrDocument{id=0, id_field_copy_that_does_not_support_in_place_update_s=0, title_s=title0, id_i=0, inplace_updatable_float=101.0, _version_=1560229605069029376, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=9487} expected:<7129> but was:<9487>
	at __randomizedtesting.SeedInfo.seed([8BCA03E3EB9108DF:39E3C39456D6527]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesInResults(TestInPlaceUpdatesDistrib.java:442)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesAgainstAllClients(TestInPlaceUpdatesDistrib.java:413)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:321)
	at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 757 lines...]
   [junit4] Suite: org.apache.lucene.search.TestShardSearching
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestShardSearching -Dtests.method=testSimple -Dtests.seed=D94E59EA03A58262 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=ar-MA -Dtests.timezone=Etc/GMT+1 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 1.21s J2 | TestShardSearching.testSimple <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: wrong hit docID expected:<11> but was:<3>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D94E59EA03A58262:E1FD7D14245656B3]:0)
   [junit4]    > 	at org.apache.lucene.util.TestUtil.assertEquals(TestUtil.java:1050)
   [junit4]    > 	at org.apache.lucene.search.TestShardSearching.assertSame(TestShardSearching.java:387)
   [junit4]    > 	at org.apache.lucene.search.TestShardSearching.testSimple(TestShardSearching.java:236)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.search.TestShardSearching_D94E59EA03A58262-001
   [junit4]   2> Feb 24, 2017 12:59:21 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {date=PostingsFormat(name=LuceneFixedGap), docid=FSTOrd50, titleTokenized=BlockTreeOrds(blocksize=128), body=PostingsFormat(name=LuceneFixedGap), title=Lucene50(blocksize=128)}, docValues:{date=DocValuesFormat(name=Lucene70), docid_intDV=DocValuesFormat(name=Lucene70), docid_int=DocValuesFormat(name=Asserting), docid=DocValuesFormat(name=Lucene70), titleTokenized=DocValuesFormat(name=Lucene70), title=DocValuesFormat(name=Asserting), body=DocValuesFormat(name=Asserting), titleDV=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1770, maxMBSortInHeap=6.364059311570613, sim=RandomSimilarity(queryNorm=false): {}, locale=ar-MA, timezone=Etc/GMT+1
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=339230568,total=439877632
   [junit4]   2> NOTE: All tests run in this JVM: [TestTopDocsCollector, TestDirectoryReader, TestGeoEncodingUtils, TestDocsWithFieldSet, TestDateSort, TestRegexpQuery, TestTermScorer, TestFrequencyTrackingRingBuffer, TestDocValuesRewriteMethod, TestPayloadsOnVectors, TestSparseFixedBitDocIdSet, TestRollingBuffer, TestPolygon2D, TestNRTThreads, TestConsistentFieldNumbers, TestMixedCodecs, TestNoMergePolicy, TestIndexWriterMaxDocs, TestNRTReaderWithThreads, TestCharTermAttributeImpl, TestFastCompressionMode, TestPositiveScoresOnlyCollector, TestSameTokenSamePosition, TestRamUsageEstimator, TestSpanNotQuery, TestEarlyTermination, TestDocIdSetBuilder, TestLongPostings, TestMultiMMap, TestLongBitSet, TestHighCompressionMode, TestSegmentReader, TestMultiTermsEnum, TestBKD, TestAllFilesHaveChecksumFooter, TestPackedInts, TestSimpleFSLockFactory, TestIndexWriterThreadsToSegments, TestRollingUpdates, Test2BSortedDocValuesOrds, TestAttributeSource, TermInSetQueryTest, TestTwoPhaseCommitTool, TestNeedsScores, TestAllFilesHaveCodecHeader, TestVersion, TestSynonymQuery, TestSimpleExplanationsOfNonMatches, TestOneMergeWrappingMergePolicy, TestDocValuesScoring, TestComplexExplanationsOfNonMatches, TestSpanCollection, TestConjunctionDISI, TestPrefixQuery, TestLongValuesSource, TestSearchForDuplicates, TestFieldsReader, TestSearchWithThreads, TestForUtil, TestDocument, TestSortedSetDocValues, TestMultiFields, TestSpansEnum, TestDirectPacked, TestReqExclBulkScorer, TestPayloads, TestHugeRamFile, TestIndexWriterCommit, TestCharacterUtils, TestCachingTokenFilter, TestLogMergePolicy, TestSpanSearchEquivalence, TestClassicSimilarity, TestStressIndexing2, TestIndexWriterLockRelease, TestIndexWriter, TestPerFieldDocValuesFormat, TestIndexWriterReader, TestIndexWriterMerging, TestBytesStore, TestGraphTokenizers, TestAddIndexes, TestShardSearching]
   [junit4] Completed [161/448 (1!)] on J2 in 15.14s, 1 test, 1 failure <<< FAILURES!

[...truncated 10859 lines...]
   [junit4] Suite: org.apache.solr.update.TestInPlaceUpdatesDistrib
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/init-core-data-001
   [junit4]   2> 562859 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 562860 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 562860 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_hs/d
   [junit4]   2> 562862 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 562863 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 562890 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 562890 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 562918 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=inplace-updates
   [junit4]   2> 562922 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 562948 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 562950 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 562950 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 562952 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 562952 WARN  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 562952 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 562953 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.c.SolrResourceLoader [null] Added 2 libs to classloader, from paths: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/lib/classes]
   [junit4]   2> 562976 WARN  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 562976 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 562985 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 562988 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 562989 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1
   [junit4]   2> 562989 WARN  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter No serviceUrl or agentId was configured, using first MBeanServer.
   [junit4]   2> 562989 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 562989 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/init-core-data-001/]
   [junit4]   2> 562989 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 562990 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=1.3251953125, floorSegmentMB=0.6796875, forceMergeDeletesPctAllowed=22.522385965353358, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.25535115980707734
   [junit4]   2> 562994 WARN  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 563022 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 563022 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 563022 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 563022 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 563023 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6a940de0
   [junit4]   2> 563023 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@525ccd4c[collection1] main]
   [junit4]   2> 563023 WARN  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 563023 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 563025 INFO  (searcherExecutor-1312-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@525ccd4c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 563025 INFO  (coreLoadExecutor-1311-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560229579125161984
   [junit4]   2> 563029 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 563032 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 563032 INFO  (Thread-612) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 563032 INFO  (Thread-612) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 563132 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ZkTestServer start zk server on port:60379
   [junit4]   2> 563144 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 563146 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema-inplace-updates.xml to /configs/conf1/schema.xml
   [junit4]   2> 563147 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 563148 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 563149 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 563150 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 563151 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 563152 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 563153 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 563154 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 563154 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 563224 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/control-001/cores/collection1
   [junit4]   2> 563225 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 563226 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12d9f759{/_hs/d,null,AVAILABLE}
   [junit4]   2> 563227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6b33fcf9{HTTP/1.1,[http/1.1]}{127.0.0.1:47004}
   [junit4]   2> 563227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server Started @566961ms
   [junit4]   2> 563227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/tempDir-001/control/data, hostContext=/_hs/d, hostPort=47004, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/control-001/cores}
   [junit4]   2> 563227 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 563227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 563227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 563227 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 563228 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-24T15:41:17.512Z
   [junit4]   2> 563230 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 563230 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/control-001/solr.xml
   [junit4]   2> 563235 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60379/solr
   [junit4]   2> 563252 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 563252 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:47004__hs%2Fd
   [junit4]   2> 563253 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.Overseer Overseer (id=97514348696240132-127.0.0.1:47004__hs%2Fd-n_0000000000) starting
   [junit4]   2> 563259 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47004__hs%2Fd
   [junit4]   2> 563260 INFO  (zkCallback-399-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 563319 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/control-001/cores
   [junit4]   2> 563319 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 563325 INFO  (OverseerStateUpdate-97514348696240132-127.0.0.1:47004__hs%2Fd-n_0000000000) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 564350 WARN  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 564351 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 564363 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 564366 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 564367 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 564367 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/control-001/cores/collection1/data/]
   [junit4]   2> 564368 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 564368 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=1.3251953125, floorSegmentMB=0.6796875, forceMergeDeletesPctAllowed=22.522385965353358, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.25535115980707734
   [junit4]   2> 564373 WARN  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 564402 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 564402 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 564403 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 564403 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 564404 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6f9cfac0
   [junit4]   2> 564404 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6a907204[collection1] main]
   [junit4]   2> 564405 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 564405 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 564405 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 564406 INFO  (searcherExecutor-1325-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6a907204[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 564406 INFO  (coreLoadExecutor-1324-thread-1-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560229580573245440
   [junit4]   2> 564412 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 564412 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 564412 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:47004/_hs/d/collection1/
   [junit4]   2> 564412 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 564412 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:47004/_hs/d/collection1/ has no replicas
   [junit4]   2> 564412 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 564415 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:47004/_hs/d/collection1/ shard1
   [junit4]   2> 564565 INFO  (coreZkRegister-1317-thread-1-processing-n:127.0.0.1:47004__hs%2Fd x:collection1 c:control_collection) [n:127.0.0.1:47004__hs%2Fd c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 564887 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 564888 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60379/solr ready
   [junit4]   2> 564888 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 564888 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 564958 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-1-001/cores/collection1
   [junit4]   2> 564959 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-1-001
   [junit4]   2> 564959 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 564960 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2f9f5107{/_hs/d,null,AVAILABLE}
   [junit4]   2> 564961 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@27cab66e{HTTP/1.1,[http/1.1]}{127.0.0.1:60027}
   [junit4]   2> 564961 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server Started @568695ms
   [junit4]   2> 564961 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/_hs/d, hostPort=60027, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-1-001/cores}
   [junit4]   2> 564961 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 564961 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 564962 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 564962 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 564962 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-24T15:41:19.246Z
   [junit4]   2> 564964 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 564964 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-1-001/solr.xml
   [junit4]   2> 564970 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60379/solr
   [junit4]   2> 564977 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 564978 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 564980 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60027__hs%2Fd
   [junit4]   2> 564981 INFO  (zkCallback-403-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 564982 INFO  (zkCallback-399-thread-3-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 564983 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 565028 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-1-001/cores
   [junit4]   2> 565028 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 565033 INFO  (OverseerStateUpdate-97514348696240132-127.0.0.1:47004__hs%2Fd-n_0000000000) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 565134 WARN  (OverseerStateUpdate-97514348696240132-127.0.0.1:47004__hs%2Fd-n_0000000000) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.Overseer Bad version writing to ZK using compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion for /collections/collection1/state.json
   [junit4]   2> 565136 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 565137 INFO  (OverseerStateUpdate-97514348696240132-127.0.0.1:47004__hs%2Fd-n_0000000000) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 565138 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 566054 WARN  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 566055 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 566065 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 566068 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 566069 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 566069 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-1-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 566070 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 566070 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=1.3251953125, floorSegmentMB=0.6796875, forceMergeDeletesPctAllowed=22.522385965353358, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.25535115980707734
   [junit4]   2> 566075 WARN  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 566119 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 566119 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 566119 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 566119 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 566121 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@60ca252a
   [junit4]   2> 566121 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@11a03448[collection1] main]
   [junit4]   2> 566122 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 566122 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 566123 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 566125 INFO  (searcherExecutor-1336-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@11a03448[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 566125 INFO  (coreLoadExecutor-1335-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560229582375747584
   [junit4]   2> 566133 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 566133 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 566133 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60027/_hs/d/collection1/
   [junit4]   2> 566133 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 566133 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:60027/_hs/d/collection1/ has no replicas
   [junit4]   2> 566133 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
   [junit4]   2> 566135 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60027/_hs/d/collection1/ shard1
   [junit4]   2> 566236 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 566286 INFO  (coreZkRegister-1330-thread-1-processing-n:127.0.0.1:60027__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 566389 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 566705 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-2-001/cores/collection1
   [junit4]   2> 566705 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-2-001
   [junit4]   2> 566706 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 566707 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1361eb57{/_hs/d,null,AVAILABLE}
   [junit4]   2> 566707 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@67200c99{HTTP/1.1,[http/1.1]}{127.0.0.1:36000}
   [junit4]   2> 566707 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server Started @570442ms
   [junit4]   2> 566707 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/_hs/d, hostPort=36000, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-2-001/cores}
   [junit4]   2> 566707 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 566707 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 566708 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 566708 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 566708 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-24T15:41:20.992Z
   [junit4]   2> 566710 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 566710 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-2-001/solr.xml
   [junit4]   2> 566716 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60379/solr
   [junit4]   2> 566722 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 566724 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 566725 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36000__hs%2Fd
   [junit4]   2> 566726 INFO  (zkCallback-403-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 566726 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 566727 INFO  (zkCallback-399-thread-3-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 566732 INFO  (zkCallback-415-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 566765 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-2-001/cores
   [junit4]   2> 566765 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 566772 INFO  (OverseerStateUpdate-97514348696240132-127.0.0.1:47004__hs%2Fd-n_0000000000) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 566873 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 566873 INFO  (zkCallback-415-thread-2-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 567776 WARN  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 567776 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 567795 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 567799 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 567800 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 567800 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-2-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 567801 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 567802 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=1.3251953125, floorSegmentMB=0.6796875, forceMergeDeletesPctAllowed=22.522385965353358, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.25535115980707734
   [junit4]   2> 567806 WARN  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 567838 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 567838 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 567839 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 567839 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 567840 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@78bea4c0
   [junit4]   2> 567840 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6009a8f5[collection1] main]
   [junit4]   2> 567841 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 567841 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 567841 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 567842 INFO  (coreLoadExecutor-1346-thread-1-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1560229584176152576
   [junit4]   2> 567844 INFO  (searcherExecutor-1347-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:36000__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6009a8f5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 567845 INFO  (coreZkRegister-1341-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 c:collection1) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 567846 INFO  (updateExecutor-412-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 567851 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 567852 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 567852 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 567852 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 567852 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:60027/_hs/d/collection1/] and I am [http://127.0.0.1:36000/_hs/d/collection1/]
   [junit4]   2> 567855 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:60027/_hs/d]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:36000__hs%252Fd&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 567859 INFO  (qtp530177572-3138) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 567864 INFO  (qtp530177572-3138) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 567864 INFO  (qtp530177572-3138) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:36000__hs%2Fd, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:36000/_hs/d","node_name":"127.0.0.1:36000__hs%2Fd","state":"down"}
   [junit4]   2> 567954 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 567954 INFO  (zkCallback-415-thread-2-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 568411 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-3-001/cores/collection1
   [junit4]   2> 568412 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-3-001
   [junit4]   2> 568412 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 568413 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@366aa916{/_hs/d,null,AVAILABLE}
   [junit4]   2> 568413 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4dd3c0f5{HTTP/1.1,[http/1.1]}{127.0.0.1:46297}
   [junit4]   2> 568413 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.Server Started @572148ms
   [junit4]   2> 568413 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/_hs/d, hostPort=46297, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-3-001/cores}
   [junit4]   2> 568414 ERROR (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 568415 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 7.0.0
   [junit4]   2> 568420 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 568420 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 568420 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-24T15:41:22.704Z
   [junit4]   2> 568422 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 568422 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-3-001/solr.xml
   [junit4]   2> 568429 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60379/solr
   [junit4]   2> 568438 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 568440 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 568441 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46297__hs%2Fd
   [junit4]   2> 568442 INFO  (zkCallback-415-thread-2-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 568442 INFO  (zkCallback-399-thread-3-processing-n:127.0.0.1:47004__hs%2Fd) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 568451 INFO  (zkCallback-422-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 568442 INFO  (zkCallback-403-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 568442 INFO  (zkCallback-409-thread-1-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 568531 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-3-001/cores
   [junit4]   2> 568531 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 568535 INFO  (OverseerStateUpdate-97514348696240132-127.0.0.1:47004__hs%2Fd-n_0000000000) [n:127.0.0.1:47004__hs%2Fd    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 568637 INFO  (zkCallback-415-thread-2-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 568637 INFO  (zkCallback-409-thread-2-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 568637 INFO  (zkCallback-422-thread-2-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 568864 INFO  (qtp530177572-3138) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:36000__hs%2Fd, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:36000/_hs/d","node_name":"127.0.0.1:36000__hs%2Fd","state":"recovering"}
   [junit4]   2> 568864 INFO  (qtp530177572-3138) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 568864 INFO  (qtp530177572-3138) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:36000__hs%252Fd&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1007
   [junit4]   2> 569366 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:60027/_hs/d/collection1/] - recoveringAfterStartup=[true]
   [junit4]   2> 569366 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync PeerSync: core=collection1 url=http://127.0.0.1:36000/_hs/d START replicas=[http://127.0.0.1:60027/_hs/d/collection1/] nUpdates=100
   [junit4]   2> 569370 INFO  (qtp530177572-3144) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 569370 INFO  (qtp530177572-3144) [n:127.0.0.1:60027__hs%2Fd c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.S.Request [collection1]  webapp=/_hs/d path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 569371 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
   [junit4]   2> 569371 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync 
   [junit4]   2> 569371 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 569371 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 569372 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 569372 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
   [junit4]   2> 569372 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
   [junit4]   2> 569372 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy No replay needed.
   [junit4]   2> 569372 INFO  (recoveryExecutor-413-thread-1-processing-n:127.0.0.1:36000__hs%2Fd x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
   [junit4]   2> 569475 INFO  (zkCallback-409-thread-2-processing-n:127.0.0.1:60027__hs%2Fd) [n:127.0.0.1:60027__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 569475 INFO  (zkCallback-422-thread-2-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 569476 INFO  (zkCallback-415-thread-2-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 569549 WARN  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 569550 INFO  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 569559 INFO  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=inplace-updates
   [junit4]   2> 569562 INFO  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema inplace-updates/1.6 with uniqueid field id
   [junit4]   2> 569564 INFO  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 569564 INFO  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-3-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 569565 INFO  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@64732321
   [junit4]   2> 569565 INFO  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=1.3251953125, floorSegmentMB=0.6796875, forceMergeDeletesPctAllowed=22.522385965353358, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.25535115980707734
   [junit4]   2> 569570 WARN  (coreLoadExecutor-1357-thread-1-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}

[...truncated too long message...]

on1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 644601 INFO  (zkCallback-422-thread-5-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46297/_hs/d/collection1/ has no replicas
   [junit4]   2> 644603 INFO  (zkCallback-422-thread-5-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46297/_hs/d/collection1/ shard1
   [junit4]   2> 644605 INFO  (zkCallback-422-thread-5-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 644703 INFO  (zkCallback-415-thread-5-processing-n:127.0.0.1:36000__hs%2Fd) [n:127.0.0.1:36000__hs%2Fd c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 644703 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 644704 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@67200c99{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 644704 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1361eb57{/_hs/d,null,UNAVAILABLE}
   [junit4]   2> 644705 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 46297
   [junit4]   2> 644705 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=172559074
   [junit4]   2> 644709 INFO  (coreCloseExecutor-1378-thread-1) [n:127.0.0.1:46297__hs%2Fd c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@41d78284
   [junit4]   2> 644774 INFO  (coreCloseExecutor-1378-thread-1) [n:127.0.0.1:46297__hs%2Fd c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 644774 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.Overseer Overseer (id=97514348696240143-127.0.0.1:46297__hs%2Fd-n_0000000003) closing
   [junit4]   2> 644775 INFO  (OverseerStateUpdate-97514348696240143-127.0.0.1:46297__hs%2Fd-n_0000000003) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:46297__hs%2Fd
   [junit4]   2> 644776 WARN  (zkCallback-422-thread-4-processing-n:127.0.0.1:46297__hs%2Fd) [n:127.0.0.1:46297__hs%2Fd    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 644776 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 644777 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4dd3c0f5{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 644778 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@366aa916{/_hs/d,null,UNAVAILABLE}
   [junit4]   2> 644779 INFO  (TEST-TestInPlaceUpdatesDistrib.test-seed#[8BCA03E3EB9108DF]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60379 60379
   [junit4]   2> 644870 INFO  (Thread-612) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60379 60379
   [junit4]   2> 644871 WARN  (Thread-612) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/aliases.json
   [junit4]   2> 	5	/solr/clusterprops.json
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 	4	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	82	/solr/overseer/collection-queue-work
   [junit4]   2> 	27	/solr/overseer/queue
   [junit4]   2> 	11	/solr/overseer/queue-work
   [junit4]   2> 	5	/solr/live_nodes
   [junit4]   2> 	5	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestInPlaceUpdatesDistrib -Dtests.method=test -Dtests.seed=8BCA03E3EB9108DF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=sk -Dtests.timezone=America/Indianapolis -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 81.8s J1 | TestInPlaceUpdatesDistrib.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: 'sanitycheck' results against client: org.apache.solr.client.solrj.impl.HttpSolrClient@5c6917db (not leader) wrong [docid] for SolrDocument{id=0, id_field_copy_that_does_not_support_in_place_update_s=0, title_s=title0, id_i=0, inplace_updatable_float=101.0, _version_=1560229605069029376, inplace_updatable_int_with_default=666, inplace_updatable_float_with_default=42.0, [docid]=9487} expected:<7129> but was:<9487>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8BCA03E3EB9108DF:39E3C39456D6527]:0)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesInResults(TestInPlaceUpdatesDistrib.java:442)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.assertDocIdsAndValuesAgainstAllClients(TestInPlaceUpdatesDistrib.java:413)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.docValuesUpdateTest(TestInPlaceUpdatesDistrib.java:321)
   [junit4]    > 	at org.apache.solr.update.TestInPlaceUpdatesDistrib.test(TestInPlaceUpdatesDistrib.java:140)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 644875 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 644875 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=854469010
   [junit4]   2> 644876 INFO  (coreCloseExecutor-1380-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@b721843
   [junit4]   2> 644921 INFO  (coreCloseExecutor-1380-thread-1) [    x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 644923 INFO  (SUITE-TestInPlaceUpdatesDistrib-seed#[8BCA03E3EB9108DF]-worker) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.update.TestInPlaceUpdatesDistrib_8BCA03E3EB9108DF-001
   [junit4]   2> Feb 24, 2017 3:42:39 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=24486, maxDocsPerChunk=110, blockSize=1019), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=24486, blockSize=1019)), sim=RandomSimilarity(queryNorm=false): {}, locale=sk, timezone=America/Indianapolis
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=4,threads=1,free=123212544,total=454557696
   [junit4]   2> NOTE: All tests run in this JVM: [TestConfigSetsAPIZkFailure, TestFieldSortValues, TestCustomDocTransformer, IndexBasedSpellCheckerTest, TestSurroundQueryParser, VMParamsZkACLAndCredentialsProvidersTest, TestWordDelimiterFilterFactory, MigrateRouteKeyTest, TestConfigReload, PKIAuthenticationIntegrationTest, TestMaxScoreQueryParser, TestHashPartitioner, SolrGangliaReporterTest, HdfsRestartWhileUpdatingTest, DeleteLastCustomShardedReplicaTest, TestReloadAndDeleteDocs, TestSerializedLuceneMatchVersion, TestTestInjection, SpellCheckCollatorTest, DistributedExpandComponentTest, TestSystemIdResolver, ReplicaListTransformerTest, CollectionTooManyReplicasTest, WordBreakSolrSpellCheckerTest, DOMUtilTest, TestCursorMarkWithoutUniqueKey, TestPostingsSolrHighlighter, ZkSolrClientTest, TestConfigsApi, SpatialFilterTest, OverseerModifyCollectionTest, SpellingQueryConverterTest, TestDocTermOrds, ClassificationUpdateProcessorFactoryTest, ConcurrentDeleteAndCreateCollectionTest, TestCopyFieldCollectionResource, ConvertedLegacyTest, CloudExitableDirectoryReaderTest, TestInPlaceUpdatesDistrib]
   [junit4] Completed [100/693 (1!)] on J1 in 83.09s, 1 test, 1 failure <<< FAILURES!

[...truncated 64523 lines...]